Search

Recent advancements in connected automated vehicles (CAVs) and reinforcement learning (RL) hold significant promise for enhancing intelligent traffic control systems. This paper conducts a systematic review of studies on RL-based urban traffic control at signalised intersections, highlighting the significant impact of CAVs on traffic control performance improvement. We first review the fundamental concepts of RL algorithms, establishing a foundational understanding for subsequent RL-based traffic control methods. We then review recent progress in RL-based traffic signal control using CV/CAV trajectory data, RL-based CAV trajectory planning, and the cooperative control of both traffic signals and CAVs at signalised intersections. Our aim is to provide researchers with a comprehensive roadmap for future research in RL-based traffic control at signalised intersections. [ABSTRACT FROM AUTHOR]

Compared to traditional traffic signal control methods, the method driven by Deep Reinforcement Learning (DRL) has shown better performance. But the problem of low sample utilization in reinforcement learning also arises. To deal with the problem, this paper presents a novel Twin Delayed Deep Deterministic Policy Gradient with Dual Buffer (TD3_DB) for traffic signal control. In the proposed framework, two experience buffers are used to store important samples and normal samples, separately, and the proportion of the two buffers is adjusted adaptively. In addition, lane pressure, describing the dynamic feature of lane traffic flow, is used for the state design of the TD3 agent, which enhances the perception of the agent toward intersections. Comprehensive experiments on different traffic flow modes has shown, the dual experience replay scheme can improve the sample utilization, and the proposed TD3_DB performs better than other methods such as original TD3, Proximal Policy Optimization (PPO), etc., effectively reducing vehicle queue length and waiting time. [ABSTRACT FROM AUTHOR]

Traffic systems play a key role in modern society. However, these systems are increasingly suffering from problems, such as congestions. A well-known way to efficiently reduce this kind of problem is to perform traffic light control intelligently through reinforcement learning (RL) algorithms. In this context, extracting relevant features from the traffic environment to support decision-making becomes a central concern. Examples of such features include vehicle counting on each queue, identification of vehicles' origins and destinations, among others. Recently, the advent of deep learning has paved to way to efficient methods for extracting some of the aforementioned features. However, the problem of identifying vehicles and their origins and destinations within an intersection has not been fully addressed in the literature, even though such information has shown to play a role in RL-based traffic signal control. Building against this background, in this work we propose a deep learning pipeline for extracting relevant features from intersections based on traffic scenes. Our pipeline comprises three main steps: (i) a YOLO-based object detector fine-tuned using the UAVDT dataset, (ii) a tracking algorithm to keep track of vehicles along their trajectories, and (iii) an origin-destination identification algorithm. Using this pipeline, it is possible to identify vehicles as well as their origins and destinations within a given intersection. In order to assess our pipeline, we evaluated each of its modules separately as well as the pipeline as a whole. The object detector model obtained 98.2% recall and 79.5% precision, on average. The tracking algorithm obtained a MOTA of 72.6% and a MOTP of 74.4%. Finally, the complete pipeline obtained an average error rate of 3.065% in terms of origin and destination counts. [ABSTRACT FROM AUTHOR]

The increasing demand for mobility in our society poses various challenges to traffic engineering, computer science in general, and artificial intelligence in particular. Increasing the capacity of road networks is not always possible, thus a more efficient use of the available transportation infrastructure is required. Another issue is that many problems in traffic management and control are inherently decentralized and/or require adaptation to the traffic situation. Hence, there is a close relationship to multiagent reinforcement learning. However, using reinforcement learning poses the challenge that the state space is normally large and continuous, thus it is necessary to find appropriate schemes to deal with discretization of the state space. To address these issues, a multiagent system with agents learning independently via a learning algorithm was proposed, which is based on estimating Q-values from k-nearest neighbors. In the present paper, we extend this approach and include transfer of experiences among the agents, especially when an agent does not have a good set of k experiences. We deal with traffic signal control, running experiments on a traffic network in which we vary the traffic situation along time, and compare our approach to two baselines (one involving reinforcement learning and one based on fixed times). Our results show that the extended method pays off when an agent returns to an already experienced traffic situation. [ABSTRACT FROM AUTHOR]

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system. [ABSTRACT FROM AUTHOR]

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

This paper proposes a reinforcement learning (RL)-based traffic control strategy integrated with attention mechanism for large-scale adaptive traffic signal control (ATSC) system. The proposed attention RL integrates attention mechanism into a multiagent RL model, namely multiagent proximal policy optimization (MAPPO), so as to enable more effective, scalable, and stable learning in complex ATSC envi)ronments. In the attention RL, decentralized policies are trained using a centrally computed critic that shares an attention model, while the attention model selects relevant intersections for each agent to estimate the global critic. This framework effectively reduces the computa)tional complexity and stabilizes the training process, enhancing the ability of RL agents to control large-scale traffic networks. The proposed control strategy is tested in both a large synthetic traffic grid and a large real-world traffic network of Yangzhou city using the microscopic traffic simulation tool, SUMO. Experimental results demonstrate that the proposed approach learns stable and sustainable policies that achieve lower congestion level and faster recovery, which outperforms other state-of-art RL-based approaches, as well as a gap-based actuated controller. [ABSTRACT FROM AUTHOR]

With the development of autonomous driving, a mixed traffic flow state composed of connected automated vehicles (CAVs) and human-driven vehicles (HVs) will last for an extended period. The abundant computing resources and CAVs with high compliance in the intelligent connected environment provide a good situation for variable speed limit control on highways, which helps even the traffic flow and improves traffic efficiency and safety. In this paper, we propose a variable speed limit control method for mixed traffic flow based on deep-reinforcement learning. First, the variable speed limit control problem is abstracted into a Markov decision process and the factors of real-time CAV penetration rates and predictions are considered in the state description. Different from variable message signs (VMS), CAVs are taken as the executive objects of the controller so that the variable speed limit control for mixed traffic flow is realized indirectly through the interaction with HVs. Next, double deep Q network (DDQN) is introduced to calculate the optimal speed limit in different states. Finally, the empirical study on US101-S proves the effectiveness of the proposed model. The results show that the variable speed limit control model based on the DDQN algorithm can effectively improve the efficiency and environmental benefits of mixed traffic flow. Moreover, the multi-objective reward function can achieve a better control effect than the single objective. Besides, the proposed model outperforms other models in this paper and predictive factors can further improve proactive control performance. In addition, with the increasing penetration of CAV, the proposed model achieves a better control effect. [ABSTRACT FROM AUTHOR]

The continuous evolution of artificial intelligence and cyber–physical systems has presented promising opportunities for optimizing traffic signal control in densely populated urban areas, with the aim of alleviating traffic congestion. One area that has garnered significant interest from both researchers and practitioners is the application of deep reinforcement learning (DRL) in traffic signal control. However, DRL-based algorithms often suffer from instability due to the dynamic nature of traffic flows. Discrepancies between the environments used for training and those encountered during deployment often lead to operational failures. Moreover, conventional DRL-based traffic signal control algorithms tend to reveal vulnerabilities when faced with unforeseen events, such as sensor failure. These challenges highlight the need for innovative solutions to enhance the robustness and adaptability of such systems. To address these pertinent issues, this paper introduces StageLight, a novel two-stage multiscale learning approach, which involves learning optimal timings on a coarse time scale in stage 1, while finetuning them on a finer time scale in stage 2. Our experimental results demonstrate StageLight's remarkable capability to generalize across diverse traffic conditions and its robustness to various sensor-failure scenarios. [ABSTRACT FROM AUTHOR]

the dynamic states of the road networks. In order to solve this problem, this paper proposed a deep reinforcement learning traffic light control strategy combined with fuzzy control, used SAC deep reinforcement learning to jointly optimize the phase selection and timing of traffic lights at two intersections, while considering multiple influencing factors, used fuzzy control to process the penalty function of SAC. The experimental results demonstrate that compared with the fixed cycle strategy, SAC control strategy and DDPG control strategy, the proposed traffic signal control strategy can obtain faster vehicle speed, and the fuel consumption and exhaust emissions of the vehicle are also improved. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) can be described as a multi-agent cooperative game. To realize cooperation, multi-agent reinforcement learning (MARL) is a significant approach, with communication being a core component. The large-scale traffic signals and the partially observable information in TSC pose a considerable challenge in finding the optimal joint control policy. This paper proposed a deep MARL model named attentional graph relations communications network (AGRCNet). Based on the Actor-Critic framework, AGRCNet designs a communication network to exchange observation information with agents to help obtain the optimal joint action, reducing the decision error caused by the partially observable condition. Specifically, through the communication network, the chain propagation of graph attention networks (GAT) and graph convolutional networks is used to expand the receptive domain of agents, improve communication efficiency and promote cooperative behavior. We simulate the traffic situation near the Nanjing Yangtze River Bridge in Simulation of Urban MObility. With a compound reward, our method performs best. Meanwhile, AGRCNet is applied to two abstract environments, and the results show that our approach can also adapt to dynamic agent relationships and is more efficient than comparison algorithms. [ABSTRACT FROM AUTHOR]

Smart agricultural systems have received a great deal of interest in recent years because of their potential for improving the efficiency and productivity of farming practices. These systems gather and analyze environmental data such as temperature, soil moisture, humidity, etc., using sensor networks and Internet of Things (IoT) devices. This information can then be utilized to improve crop growth, identify plant illnesses, and minimize water usage. However, dealing with data complexity and dynamism can be difficult when using traditional processing methods. As a solution to this, we offer a novel framework that combines Machine Learning (ML) with a Reinforcement Learning (RL) algorithm to optimize traffic routing inside Software-Defined Networks (SDN) through traffic classifications. ML models such as Logistic Regression (LR), Random Forest (RF), k-nearest Neighbours (KNN), Support Vector Machines (SVM), Naive Bayes (NB), and Decision Trees (DT) are used to categorize data traffic into emergency, normal, and on-demand. The basic version of RL, i.e., the Q-learning (QL) algorithm, is utilized alongside the SDN paradigm to optimize routing based on traffic classes. It is worth mentioning that RF and DT outperform the other ML models in terms of accuracy. Our results illustrate the importance of the suggested technique in optimizing traffic routing in SDN environments. Integrating ML-based data classification with the QL method improves resource allocation, reduces latency, and improves the delivery of emergency traffic. The versatility of SDN facilitates the adaption of routing algorithms depending on real-time changes in network circumstances and traffic characteristics. [ABSTRACT FROM AUTHOR]

As urban areas continue to expand, traffic congestion has emerged as a significant challenge impacting urban governance and economic development. Frequent regional traffic congestion has become a primary factor hindering urban economic growth and social activities, necessitating improved regional traffic management. Addressing regional traffic optimization and control methods based on the characteristics of regional congestion has become a crucial and complex issue in the field of traffic management and control research. This paper focuses on the macroscopic fundamental diagram (MFD) and aims to tackle the control problem without relying on traffic determination information. To address this, we introduce the Q-learning (QL) algorithm in reinforcement learning and the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning. Subsequently, we propose the MFD-QL perimeter control model and the MFD-DDPG perimeter control model. We conduct numerical analysis and simulation experiments to verify the effectiveness of the MFD-QL and MFD-DDPG algorithms. The experimental results show that the algorithms converge rapidly to a stable state and achieve superior control effects in optimizing regional perimeter control. [ABSTRACT FROM AUTHOR]

Tackling traffic signal control through multi-agent reinforcement learning is a widely-employed approach. However, current state-of-the-art models have drawbacks: intersections optimize their own local rewards and cause traffic to waste time and fuel with a start-stop mode at each intersection. They also lack information sharing among intersections and their specialized policy hinders the ability to adapt to new traffic scenarios. To overcome these limitations, This work presents a centralized collaborative graph network (CCGN) with the core objective of a signal-free corridor once the traffic flows have waited at the entry intersection of the traffic intersection network on either side, the subsequent intersection gives the open signal as the traffic flows arrive. CCGN combines local policy networks (LPN) and global policy networks, where LPN employed at each intersection predicts actions based on Transformer and Graph Convolutional Network (GCN). In contrast, GPN is based on GCN and Q-network that receives the LPN states, traffic flow and road information to manage intersections to provide a signal-free corridor. We developed the Deep Graph Convolution Q-Network (DGCQ) by combining Deep Q-Network (DQN) and GCN to achieve a signal-free corridor. DGCQ leverages GCN's intersection collaboration and DQN's information aggregation for traffic control decisions Proposed CCGN model is trained on the robust synthetic traffic network and evaluated on the real-world traffic networks that outperform the other state-of-the-art models. [ABSTRACT FROM AUTHOR]

The emerging technologies of connected and automated vehicles (CAVs) and deep reinforcement learning (DRL) provide innovative methods and have a great potential for developing new solutions to improve the efficiency of several intersection systems. Based on the multisource data collected from the transportation environments, CAVs with the cooperative adaptive cruise control (CACC) system could merge into platoons and traverse the intersection quickly and smoothly. Meanwhile, the traffic information about the CAVs enables intelligent traffic signal controls with the help of DRL technologies. This research investigates the performance of a state-shared multiagent deep reinforcement learning (MADRL) controlled signal corridor with platooning-based CAVs. A corridor with seven intersections from the Ingolstadt Traffic Scenario (InTAS) in Germany is selected as a case study. The state information is shared between neighboring intersections to overcome the partial information observation of the decentralized agents in the MADRL framework. A platooning framework with specific CACC systems for the leading and following vehicles is proposed. Results indicate that the state-shared MADRL with CAV platoons could significantly decrease the total waiting time, average queue length, and total CO2 emission of the corridor by 80%, 73%, and 54%, respectively, which could be beneficial in further improving the intersection efficiency, designing future intersections, and cooperating signals and CAVs platoons. [ABSTRACT FROM AUTHOR]

Vehicular Ad Hoc Network (VANETs) need methods to control traffic caused by a high volume of traffic during day and night, the interaction of vehicles, and pedestrians, vehicle collisions, increasing travel delays, and energy issues. Routing is one of the most critical problems in VANET. One of the machine learning categories is reinforcement learning (RL), which uses RL algorithms to find a more optimal path. According to the feedback they get from the environment, these methods can affect the system through learning from previous actions and reactions. This paper provides a comprehensive review of various methods such as reinforcement learning, deep reinforcement learning, and fuzzy learning in the traffic network, to obtain the best method for finding optimal routing in the VANET network. In fact, this paper deals with the advantages, disadvantages and performance of the methods introduced. Finally, we categorize the investigated methods and suggest the proper performance of each of them. [ABSTRACT FROM AUTHOR]

We studied the ability of deep reinforcement learning and self-organizing approaches to adapt to dynamic complex systems, using the applied example of traffic signal control in a simulated urban environment. We highlight the general limitations of deep learning for control in complex systems, even when employing state-of-the-art meta-learning methods, and contrast it with self-organization-based methods. Accordingly, we argue that complex systems are a good and challenging study environment for developing and improving meta-learning approaches. At the same time, we point to the importance of baselines to which meta-learning methods can be compared and present a self-organizing analytic traffic signal control that outperforms state-of-the-art meta-learning in some scenarios. We also show that meta-learning methods outperform classical learning methods in our simulated environment (around 1.5–2× improvement, in most scenarios). Our conclusions are that, in order to develop effective meta-learning methods that are able to adapt to a variety of conditions, it is necessary to test them in demanding, complex settings (such as, for example, urban traffic control) and compare them against established methods. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning has found great successes in addressing many challenging control problems; however, real-world implementations are still scarce if not non-existent. This is primarily due to three main challenges pertaining to the implementation, the stability, the optimal settings, and a lack of knowledge on methods that can be applied to field settings. This research attempts to address these issues with an adaptive simulation-based control framework proposed specifically for the training and evaluation. The control framework has implemented simulation models to conduct an extensive sensitivity analysis on the effects of key design variables, including rewarding schemes, state spaces, and model training parameters. The feasibility of transfer learning as a training strategy is also studied on scenarios with different layouts and different driver behavior models. Complex scenarios are also evaluated and used as test cases, including multiphase ring-and-barrier control and multi-intersection control. The research has contributed a significant amount of evidence on several critical design and implementation-related questions such as input representation, data (technology) requirements, training methods, and model transferability. [ABSTRACT FROM AUTHOR]

• A Causal-Inference (CI) model is designed for the non-stationary multi-agent environment. • Combining with Multi-Agent learning, a CI-MA algorithm is proposed for traffic signal control. • Different granularity of traffic information is fused for feature representation. • A representation loss function and MA loss function are designed for joint optimization. • Experiments show that CI-MA algorithm outperforms the state-of-the-art algorithms. A primary challenge in multi-agent reinforcement learning for traffic signal control is to produce effective cooperative traffic-signal policies in non-stationary multi-agent traffic environments. However, each agent suffers from its local non-stationary traffic environment caused by the time-varying traffic-signal policies of adjacent agents; At the same time, different agents also produce time-varying traffic-signal policies, which further results in the non-stationarity of the whole traffic environment, so these produced traffic-signal policies may be ineffective. In this work, we propose a Causal Inference Multi-Agent reinforcement learning (CI-MA) algorithm, which can alleviate the non-stationarity of multi-agent traffic environments from both feature representation and optimization, eventually helps to produce effective cooperative traffic-signal policies. Specifically, a Causal-Inference (CI) model is first designed to reason about and tackle the non-stationarity of multi-agent traffic environments by both acquiring feature representation distributions and deriving variational lower bounds (i.e., objective functions); And then, based on the designed CI model, we propose a CI-MA algorithm, in which the feature representations are acquired from the non-stationarity of multi-agent traffic environments at both task level and timestep level, the acquired feature representations are used to produce cooperative traffic-signal policies and Q-values for multiple agents; Finally the corresponding objective functions optimize the whole algorithm from both causal inference and multi-agent reinforcement learning. Experiments are conducted in different non-stationary multi-agent traffic environments. Results show that CI-MA algorithm outperforms other state-of-the-art algorithms, and demonstrate that the proposed algorithm trained in synthetic-traffic environments can be effectively transferred to both synthetic- and real-traffic environments with non-stationarity. [ABSTRACT FROM AUTHOR]

This manuscript presents an Adam optimization-based Deep Reinforcement Learning model for Mixed Traffic Flow control (ADRL-MTF), so as to guide Connected and Autonomous vehicle's (CAV) longitudinal trajectory on a typical urban roadway with signal-controlled intersections. Two improvements are made when compared with prior literatures. First, the common assumptions to simplify the problem solving, such as dividing a vehicle trajectory into several segments with constant acceleration/deceleration, are avoided, to improve the modeling realism. Second, built on the efficient Adam Optimization and Deep Q-Learning, the proposed model avoids the enumeration of states and actions, and is computational efficient and suitable for real time applications. The mixed traffic flow dynamic is firstly formulated as a finite Markov decision process (MDP) model. Due to the discretization of time, space and speed, this MDP model becomes high-dimensional in state, and is very challenging to solve. We then propose a temporal difference-based deep reinforcement learning approach, with ε -greedy for exploration-exploitation balance. Two neural networks are developed to replace the traditional Q function and generate the targets in the Q-learning update. These two neural networks are trained by the Adam optimization algorithm, which extends stochastic gradient descent and considers the second moments of the gradients, and is thus highly computational efficient and has lower memory requirements. The proposed model is shown to reduce fuel consumption by 7.8%, which outperforms a prior benchmark model based on Monte Carlo Tree Search. The model's runtime efficiency and stability are tested, and the sensitivity analysis is also performed. [ABSTRACT FROM AUTHOR]

To contend traffic congestion on urban networks, existing studies have made great efforts to develop traffic-responsive signal timing algorithms in the last decade. More recently, as an alternative to conventional model-based algorithms, machine learning-based methods have been tested on traffic light timing problems and show promising potentials. However, many researchers and practitioners still questioned the feasibility and applicability of adopting machine learning techniques in the ATSC domain. One of the reasons is that these methods assumed flawless detectors and heavily relied on simulators for training and evaluations. To address such a critical concern, this article customizes a Deep Q-learning Learning (DQL) method to optimize traffic signal timings at urban intersections, where the partial observations from identity-based detectors are inputs, and the green splits are outputs. A simulation-free data-driven prediction model is also developed to train the DQL with reduced computational time. Then the machine learning-based methods are evaluated on a real-world case with Automatic Number-Plate Recognition (ANPR) data. Experiments show the proposed data-driven model can predict the traffic state in limited computational time, and the DQL algorithm is 3.9% better than the field experiment performance from the adaptive control system, SCOOT, and 22% better than the time-of-day plan by SYNCHRO. The results indicate the DQL methods can only yield marginal improvement with restrictive input and output settings in congested traffic flow in comparison to the conventional adaptive method. [ABSTRACT FROM AUTHOR]

DRL has gained wild applications in the field of urban transportation signal control. However, the existing DRL traffic signal control researches only use the traditional deep neural network, and its perception ability is limited in complex traffic scenarios. In addition, as one of the three elements of reinforcement learning, it also needs to design the traffic state carefully and manually in the existing researches. Therefore, this paper proposed a DRL traffic signal control algorithm based on attention mechanism. By introducing the attention mechanism, the neural network could automatically pay attention to the important state components to enhance the perception ability of the network, improve the signal control effect, and reduce the difficulty of state vector design. Experimental results on SUMO platform show that compared with the three classical signal control algorithms, only using a simple traffic state, the proposed algorithm has the best performance in average waiting time and travel time under the condition of low and high traffic flow at single intersections and multiple intersections. [ABSTRACT FROM AUTHOR]

Traffic lights play a vital role in the operation of the traffic network. However, with the rapid development of traffic, roads are becoming more and more complex, and vehicles are becoming more and more numerous, which leads to the increasing pressure of traffic lights scheduling, but the regulation ability is becoming weaker and weaker. In order to solve this problem, this paper established the convergence trace source (CTS) scheme. This scheme used the traffic light, the main object of traffic diversion, as an agent for reinforcement learning to optimize its ability to control traffic diversion. It comprehensively analyzed the congestion situation of the road network by constructing the congestion chain and congestion ring, and used the traffic light phase and its timing data to achieve the comprehensive judgment of the object state of the traffic light agent. This scheme designed the traffic light queue length algorithm, and used the digitization of congestion as an agent reward to evaluate the optimization effect. This paper used the SUMO simulation environment for experiments, designed and compared the average queue length at the intersection of the traffic optimization index. Finally, the average queue length at the intersection of this scheme is increased by 40% compared with the original data. [ABSTRACT FROM AUTHOR]

The Internet of Things has a large number of terminal devices and a wide range of deployment. This feature has higher requirements for reliable transmission and parallel connection, and is more prone to network congestion resulting in the loss of power information flow and excessive delay. The traditional congestion control algorithm is not suitable for the high concurrent mass heterogeneous iot terminal access control and mass information flow storage control. In this paper, the Transformer BBR algorithm is proposed, which is a congestion control algorithm based on BBR deep reinforcement learning combined with Transformer's excellent long-term prediction ability. In the BBR algorithm bandwidth detection stage, transformer model is used as an agent to detect the network state and map it to the congestion window, so as to find the network changing state in time and make the corresponding decision action. Firstly, the congestion control strategy is learned in the simulation network environment, and then the efficiency is verified in the simulation environment. Finally, the experiment shows that the algorithm can reduce the delay while ensuring good bandwidth performance, and is superior to the main application algorithms in network throughput. [ABSTRACT FROM AUTHOR]

The paper is devoted to scheduling in multiagent systems in the framework of the Flatland 3 competition. The main aim of this competition is to develop an algorithm for the effective control of dense traffic in complex railroad networks according to a given schedule. The proposed solution is based on reinforcement learning. To adapt this method to the particular scheduling problem, a novel approach based on structuring the reward function that stimulates an agent to adhere to its schedule was developed. The architecture of the proposed model is based on a multiagent version of centralized critic with proximal policy optimization (PPO) learning. In addition, a curriculum learning strategy was developed and implemented. This allowed the agent to cope with each level of complexity on time and train the model in more difficult conditions. The proposed solution won first place in the Flatland 3 competition in the reinforcement learning track. [ABSTRACT FROM AUTHOR]

• An algorithm based on heterogeneous GNNs is proposed for feature fusion. • A mutual-information loss function is proposed for training. • Based on above methods, an IHA-MDGI algorithm is proposed for traffic signal control. • Experiments show the IHA-MDGI algorithm outperforms the state-of-the-art algorithms. Adaptive traffic signal control (ATSC) facilitates alleviating traffic congestion. Multi-agent deep reinforcement learning (MDRL) is a new promising algorithm for ATSC, and Graph Neural Networks (GNNs) further promote its learning ability. However, there are some drawbacks in the state-of-the-art MDRL algorithms. (1) These algorithms cannot effectively fuse diverse heterogeneous information of traffic networks due to adopting homogeneous GNNs; (2) These algorithms cannot be effectively trained due to merely adopting MDRL loss functions. In this paper, we propose an Inductive Heterogeneous graph Attention-based Multi-agent Deep Graph Infomax (IHA-MDGI) algorithm for ATSC. The proposed IHA-MDGI algorithm conducts both feature fusion via a proposed Inductive Heterogeneous graph Attention (IHA) algorithm and training via a proposed Multi-agent Deep Graph Infomax (MDGI) framework. Specifically, (1) Unlike the algorithms which adopt homogeneous GNNs, in the IHA algorithm, heterogeneous GNNs are designed to fuse both heterogeneous structural information and heterogeneous features of traffic networks, which aims to acquire heterogeneous information embeddings of traffic networks. (2) In the MDGI framework, the acquired embeddings are used to calculate the signal-control policies and Q -value for each agent, and then a mutual-information loss function is designed, which combines with the MDRL loss function to jointly train the whole algorithm. The designed mutual-information loss function focuses on maximizing mutual information between input (i.e., heterogeneous information embeddings) and output (i.e., Q -value), which can produce cooperative signal-control policies and maximize Q -value. We conduct the experiments in both real-traffic and synthetic-traffic networks under the time-varying traffic flows, and the results demonstrate that IHA-MDGI algorithm outperforms the state-of-the-art MDRL algorithms about multiple metrics. [ABSTRACT FROM AUTHOR]

The increasing traffic demand in urban areas frequently causes traffic congestion, which can be managed only through intelligent traffic signal controls. Although many recent studies have focused on reinforcement learning for traffic signal control (RL-TSC), most have focused on improving performance from an intersection perspective, targeting virtual simulation. The performance indexes from intersection perspectives are averaged by the weighted traffic flow; therefore, if the balance of each movement is not considered, the green time may be overly concentrated on the movements of heavy flow rates. Furthermore, as the ultimate purpose of traffic signal control research is to apply these controls to the real-world intersections, it is necessary to consider the real-world constraints. Hence, this study aims to design RL-TSC considering real-world applicability and confirm the appropriate design of the reward function. The limitations of the detector in the real world and the dual-ring traffic signal system are taken into account in the model design to facilitate real-world application. To design the reward for balancing traffic movements, we define the average delay weighted by traffic volume per lane and entropy of delay in the reward function. Model training is performed at the prototype intersection for ensuring scalability to multiple intersections. The model after prototype pre-training is evaluated by applying it to a network with two intersections without additional training. As a result, the reward function considering the equality of traffic movements shows the best performance. The proposed model reduces the average delay by more than 7.4% and 15.0% compared to the existing real-time adaptive signal control at two intersections, respectively. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning provides a new approach to solving complex signal optimization problems at intersections. Earlier studies were limited to traditional traffic detection techniques, and the obtained traffic information was not accurate. With the advanced in technology, we can obtain highly accurate information on the traffic states by advanced detector technology. This provides an accurate source of data for deep reinforcement learning. There are many intersections in the urban network. To successfully apply deep reinforcement learning in a situation closer to reality, we need to consider the problem of extending the knowledge gained from the training to new scenarios. This study used advanced sensor technology as a data source to explore the variation pattern of state space under different traffic scenarios. It analyzes the relationship between the traffic demand and the actual traffic states. The model learned more from a more comprehensive state space of traffic. This model was successful applied to new traffic scenarios without additional training. Compared our proposed model with the popular SAC signal control model, the result shows that the average delay of the DQN model is 5.13 s and the SAC model is 6.52 s. Therefore, our model exhibits better control performance. [ABSTRACT FROM AUTHOR]

In order to improve the traffic capacity of the intersection and shorten the traffic delay, a new urban intelligent traffic signal coordination control system based on machine learning is proposed in this paper. The hardware of the system is designed, including annunciator, vehicle detection module, traffic flow statistics module and signal control module. The reinforcement learning algorithm in machine learning is used to train and learn the traffic vehicle data, so as to extract the spatial characteristics of the traffic environment state. Under the constraint of the minimum loss function, the training objective is solved and the output result of the optimal coordinated control of traffic signals is output. The experimental results show that compared with the traditional control system, the maximum traffic capacity of the intersection under the control of this system is 2400 vehicles/hour, and the maximum traffic delay is no more than 30s. [ABSTRACT FROM AUTHOR]

Routing technologies have long been available in many automobiles and smart phones, but the nearly random nature of traffic on road networks has always encouraged further efforts to improve the reliability of navigation systems. Given the networks' uncertainty, an adaptive dynamic route selection model based on reinforcement learning is proposed. In the proposed method, the Markov decision process (MDP) is used to train simulated agents in a network so that they are able to make independent decisions under random conditions and, accordingly, determine the set of routes with the shortest travel time. The aim of the research was to integrate the MDP with a multi-nomial logit model (a widely used stochastic discrete-choice model) to improve finding the stochastic shortest path by computing the probability of selecting an arc from several interconnected arcs based on observations made at the arc location. The proposed model, tested with real data from part of the road network in Isfahan, Iran, and the results obtained demonstrated its good performance under 100 randomly applied stochastic scenarios. [ABSTRACT FROM AUTHOR]

Adaptive traffic signal control (ATSC) is an effective method to reduce traffic congestion in modern urban areas. Many studies adopted various approaches to adjust traffic signal plans according to real-time traffic in response to demand fluctuations to improve urban network performance (e.g., minimise delay). Recently, learning-based methods such as reinforcement learning (RL) have achieved promising results in signal plan optimisation. However, adopting these self-learning techniques in future traffic environments in the presence of connected and automated vehicles (CAVs) remains largely an open challenge. This study develops a real-time RL-based adaptive traffic signal control that optimises a signal plan to minimise the total queue length while allowing the CAVs to adjust their speed based on a fixed timing strategy to decrease total stop delays. The highlight of this work is combining a speed guidance system with a reinforcement learning-based traffic signal control. Two different performance measures are implemented to minimise total queue length and total stop delays. Results indicate that the proposed method outperforms a fixed timing plan (with optimal speed advisory in a CAV environment) and traditional actuated control, in terms of average stop delay of vehicle and queue length, particularly under saturated and oversaturated conditions. [ABSTRACT FROM AUTHOR]

Increasing urban traffic and congestion have led to significant issues such as rising air pollution and wasted time, highlighting the need for an intelligent traffic light control (TLC) system to minimize vehicle waiting times. This paper presents a novel TLC system that leverages the Internet of Things (IoT) for data collection and employs the random forest algorithm for preprocessing and feature extraction. A deep belief network predicts future traffic conditions, and a support vector regression network is integrated to enhance prediction accuracy. Additionally, the traffic light control strategy is optimized using reinforcement learning. The proposed method is evaluated through two different scenarios. The first scenario is compared with fixed-time control and the double dueling deep neural network (3DQN) methods. The second scenario compares it with the SVM, KNN, and MAADAC approaches. Simulation results demonstrate that the proposed method significantly outperforms these alternative approaches, showing substantial improvements in average vehicle waiting times by more than 20%, 32%, and 45%, respectively. Using a deep belief network, supplemented by support vector regression, ensures high precision in forecasting traffic patterns. Furthermore, the reinforcement learning-based optimization of the traffic light control strategy effectively adapts to changing traffic conditions, providing superior traffic flow management. The results indicate that the proposed system can substantially reduce traffic congestion and improve urban traffic flow. [ABSTRACT FROM AUTHOR]

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM). • Macro–Micro Control Integration: We tackled the challenge of harmonizing macroscopic (Variable Speed Limit - VSL) and microscopic (Connected and Automated Vehicles - CAVs) controls to enhance both granular and overarching traffic efficiency. Our proposed method, the Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME), cooperates with these control methods, paving the way for synchronized and efficient traffic management. • We integrated the Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework to amplify the decision-making capabilities of vehicles and developed an advanced evolutionary strategy for VSL control to facilitate real-time speed limit adjustments. • We proposed a multi-objective reward function, considering both micro and macro efficiency and safety considerations, to promote optimized traffic flow and enhance overall control performance. [ABSTRACT FROM AUTHOR]

Perimeter control is a traffic management approach aimed at regulating vehicular accumulation within urban regional networks by managing flows on all border-crossing roads. Methods based on the macroscopic fundamental diagram (MFD) fall short in providing specific metering for individual roads. Recent advancements in the cell transmission model (CTM) have attempted to address this limitation but are hindered by their reliance on centralized control, which requires the availability of full information and authority over traffic generation sites. Our study proposes an innovative decentralized, game-theoretical framework for perimeter control to address these practical challenges. It is structured around two key groups of agents: perimeter agents, tasked with managing border roads, and interior agents, focused on traffic within generation sites. The framework also incorporates mechanisms for interactions between these agents and the road network, aiming to optimize their individual utilities. Additionally, we have developed a multi-agent reinforcement learning (RL) algorithm, extending the mean-field theory concept, to address the complexity of simultaneous learning by multiple agents. • Developed a perimeter control method for heterogeneous decision-makers. • Applied the macroscopic fundamental diagram concept to simplify perimeter control. • Introduced innovative reinforcement learning techniques for problem-solving. • Demonstrated superior performance using real-world traffic data. [ABSTRACT FROM AUTHOR]

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Traffic Signal Control (TSC) plays a crucial role in mitigating congestion. The extensive integration of Deep Reinforcement Learning (DRL) into TSC, fueled by the advancements in artificial intelligence, has yielded remarkable efficacy. Nevertheless, prevailing DRL models are typically developed within homogeneous road networks, characterized by identical intersection types and relatively balanced traffic flow patterns. This predominant approach often results in an oversight of the impact of intersection heterogeneity, encompassing variations in intersection geometric structure, traffic demands, signal timing design, and other pertinent factors, on action selections by researchers. This paper addresses the aforementioned gap by introducing a value-decomposition-based spatiotemporal graph attention multi-agent DRL model (MARL_SGAT) that expressly accommodates intersection heterogeneity. Specifically, a heterogeneous correlation index is designed using road network structural parameters and serves to formulate a heterogeneous reward function, enabling the quantification of action execution improvements in diverse road network environments. To address the intricate spatiotemporal features of traffic flows within the heterogeneous road network, the model strategically selects and incorporates these features during the formulation by leveraging a spatiotemporal graph attention network. On these bases, double dual networks are introduced to convert the individual-global-max constraint of action selection into a value range constraint of the action advantage function, thereby facilitating model learning. Simulation results showcase the superior performance of the MARL_SGAT algorithm when compared to seven baseline algorithms. Specifically, the algorithm manifests in reduced average vehicle delays, decreased number of stops, and elevated travel speeds within the controlled road network. These advantages are particularly conspicuous in heterogeneous road network environments. • MARL method for collaborative control of heterogeneous intersections is proposed. • A correlation index is formulated to quantify intersection topology variations. • A novel reward function is designed to offer more precise evaluation method. • The double dual networks simplify the learning process for Q value function. • Performance of the proposed model is validated using actual data set. [ABSTRACT FROM AUTHOR]

Traffic signal control has always been a hot topic in the field of intelligent transportation. With the increasing complexity of urban traffic conditions due to urbanization, how to develop effective scheduling strategies to adapt to the changing traffic demands has become a key problem in current intelligent transportation. In light of this, this paper focuses on the traffic signal control problem at intersections and proposes a composite intelligent traffic signal control model based on heterogeneous graph neural networks with dual attention mechanisms and deep reinforcement learning. For the first time, the model incorporates the dual attention mechanism in graph neural networks into the traffic signal control, integrating graph neural networks with deep reinforcement learning techniques and traffic intersection scenarios. This allows for the construction of traffic condition models and the scheduling control of traffic resources, catering to the perception and decision-making needs in complex traffic environments. Firstly, the graph relationship representation of intersection resources is established, constructing the graph information structure for traffic flow and signal states. Then, a heterogeneous graph neural network is designed, incorporating both node-level and semantic-level dual attention mechanisms to characterize the traffic state and explore the relationships, enabling the extraction of explicit and implicit information in traffic intersections. Lastly, a deep reinforcement learning algorithm that combines Double Deep Q-Network (DDQN) and Dueling DQN is implemented to improve the algorithm's generalization and execution efficiency, enhancing the adaptability and stability of traffic signal scheduling in complex environments. Simulation tests are conducted on the SUMO simulation platform using real-world application datasets. Compared to four other similar traffic control model, the proposed model demonstrates performance advantages of more than 13% in terms of average reward, average delay, queue length, and waiting time. This validates the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

Conventional platooning control algorithms that improve traffic flow require complex computations and are not well-suited for real-time operations. Moreover, prior research has primarily concentrated on idealized settings, such as isolated intersections or corridors that solely consist of fully connected automated vehicles (CAVs), making it challenging to be applied to real-world traffic systems. To overcome these challenges, this paper aims to design an innovative learning framework based on arrival timing vectors (ATVs) and traffic signal synchronization for platooning control that can improve traffic throughput and reduce fuel consumption through four basic platoon maneuvers: split, acceleration, deceleration, and no-op. We integrate reinforcement learning (RL) with neural networks (NNs) to model non-linear relationships between inputs and outputs for a complex application. The utilization of ATVs as inputs in our proposed framework furnishes an agent with ample information to facilitate a better comprehension of the actions that need to be taken for particular states. Additionally, our proposed DRL-based platooning control collaborates with traffic signal synchronization across adjacent intersections to optimize traffic efficiency in a further step. Numerical experiments have been conducted on both corridor and network scenarios, demonstrating the positive effects of synchronization and DRL-based platooning control on average delay and fuel consumption. Various DRL-based methods have been compared to demonstrate the fast convergence and improved mobility of our proposed algorithm. Furthermore, our algorithm shows robust performance in mixed-traffic environments of CAV and human-driven vehicles. • A new DRL framework using deep learning to manipulate vehicle platoon behaviors. • The approach integrates platooning control with traffic signal synchronization. • A comparison shows the 2DSARSA algorithm outperforms other popular DRL methods. • Case studies showed DRL-based control reduce delay and fuel use in different networks. [ABSTRACT FROM AUTHOR]

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%). • Mixed AV zone. • Online operations of joint order matching and AV coordination. • Offline deep reinforcement learning framework. • Bisimulation-guided exploration. • A scalable approximate algorithm. • Numerical studies in realistic Wangjing network. [ABSTRACT FROM AUTHOR]

With the significant rise in demand for network utilization, such as data transmission and device-to-device (D2D) communication, fifth-generation (5G) networks have been proposed to fill the demand. Deploying 5G enhances the utilization of network channels and allows users to exploit licensed channels in the absence of primary users (PUs). In this paper, a hybrid route selection mechanism is proposed, and it allows the central controller (CC) to evaluate the route map proactively in a centralized manner for source nodes. In contrast, source nodes are enabled to make their own decisions reactively and select a route in a distributed manner. D2D communication is preferred, which helps networks to offload traffic from the control plane to the data plane. In addition to the theoretical analysis, a real testbed was set up for the proof of concept; it was composed of eleven nodes with independent processing units. Experiment results showed improvements in traffic offloading, higher utilization of network channels, and a lower interference level between primary and secondary users. Packet delivery ratio and end-to-end delay were affected due to a higher number of intermediate nodes and the dynamicity of PU activities. [ABSTRACT FROM AUTHOR]

Traffic congestion alleviation around intersections has been a growing challenge, and a competent traffic signal control scheme plays a pivotal role in intelligent transportation systems. Recent studies using deep reinforcement learning techniques have shown promising results for traffic signal control, but they only focus on extracting features from traffic conditions of isolated or adjacent intersections. In this work, we employed navigation information for traffic signal control, greatly enriching the features for traffic signal control with deep reinforcement learning. In addition, we are the first to propose a novel scheme DeepNavi to exploit the temporal-spatial relations from numerous navigation routes and extract dynamic real-time and future traffic features. We tested our scheme on a challenging real-world traffic dataset with 16 intersections in a residential district of Hangzhou, China. Extensive experiments were conducted and the results demonstrated that our DeepNavi scheme achieves superior performance over five popular and state-of-the-art baseline methods on different metrics, including queue length, speed, travel time and accumulative waiting time. In addition, with our method, vehicles suffer the least red lights and enjoy the most green waves, which further validates that our scheme greatly relieves the congestion and provides excellent experience for drivers. Simulations with different penetration levels of navigation routes showed that even with only part of navigation routes available in the traffic network, our scheme can obtain superior performance, further demonstrating the effectiveness and feasibility of DeepNavi. [ABSTRACT FROM AUTHOR]

Urban traffic signal control can widely use deep reinforcement learning (DRL) technique. However, in existing researches, most DRL agents only use the current traffic state to make decisions and have limited control effects when the traffic flow changes greatly. Aiming at the problem, this paper proposed a state prediction based deep reinforcement learning algorithm for traffic signal control. The algorithm used one-hot coding to design a concise and efficient traffic state, and then used a Long Short-Term Memory（LSTM） to predict the future state. The agent made optimal decisions based on the current state and the predicted state. The experimental results on the simulation platform SUMO show that compared with three typical signal control algorithms, the proposed algorithm has the best performance in terms of average waiting time, travel time, fuel consumption, CO2 emissions and cumulative reward both in a single intersection and multiple intersections under different flow conditions. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) is a challenging issue in managing an urban transportation system. A fixed time TSC is easy to implement but has drawbacks in such measures as flow rate, waiting time, and traffic density. The situation gets worse when the arrival rates of vehicles periodically change over time, which is usual in most urban cities. We propose adaptive reinforcement learning (RL) to manage TSC with varying vehicle arrival rates. Our objectives are to improve the averages of flow rate and waiting time and reduce the wasteful green light problem by considering the vehicle densities of the current lane and the downstream directions. Experiments were conducted by Simulation of Urban MObility (SUMO) under three traffic layouts and various vehicle arrival rates. The proposed method not only reduced on average traffic density, waiting time, and queue length, but also increased the average flow rate and average speed, relative to the other algorithms tested. [ABSTRACT FROM AUTHOR]

In recent years, the growing development of Connected Autonomous Vehicles (CAV), Intelligent Transport Systems (ITS), and 5G communication networks have led to the advent of Autonomous Intersection Management (AIM) systems. AIMs present a new paradigm for CAV control in future cities, taking control of CAVs in scenarios where cooperation is necessary and allowing safe and efficient traffic flows, eliminating traffic signals. So far, the development of AIM algorithms has been based on basic control algorithms, without the ability to adapt or keep learning new situations. To solve this, in this paper we present a new advanced AIM approach based on end-to-end Multi-Agent Deep Reinforcement Learning (MADRL) and trained using Curriculum through Self-Play, called advanced Reinforced AIM (adv.RAIM). adv.RAIM enables the control of CAVs at intersections in a collaborative way, autonomously learning complex real-life traffic dynamics. In addition, adv.RAIM provides a new way to build smarter AIMs capable of proactively controlling CAVs in other highly complex scenarios. Results show remarkable improvements when compared to traffic light control techniques (reducing travel time by 59% or reducing time lost due to congestion by 95%), as well as outperforming other recently proposed AIMs (reducing waiting time by 56%), highlighting the advantages of using MADRL. [ABSTRACT FROM AUTHOR]

Typical reinforcement learning (RL) requires a huge amount of data before achieving an acceptable result, and its performance can be rather poor during initial interacting process. Sample inefficiency and cold-start phenomenon of RL limits its feasibility in a range of real-world applications such as traffic signal control (TSC). On the other hand, a large amount of data on TSC can be accumulated by various model-based controllers (MBCs) rooted in traffic engineering. In this context, we propose a new RL approach which can avoid the appearance of cold starts by taking advantage of MBC experiences. First, three frameworks of joint utilization of RL and MBC are summarized for TSC, and staged framework is considered to have the edge over the other two. Then, a staged noisy-net prioritized dueling double deep Q-network (NPDD-DQN) is described in detail for TSC, where MBC experiences are used in both pre-training and online training processes. Experimental evaluation demonstrates that staged NPDD-DQN can achieve a boost in initial performance as compared to pure NPDD-DQN that does not utilize any control experiences, and learn to improve final performance beyond the underlying MBC. The effectiveness of the proposed method opens up the possibility of real implementation of RL in TSC. [ABSTRACT FROM AUTHOR]

This paper proposes a distributed deep reinforcement learning algorithm for the traffic light control problem, which consists of local learning and global consensus. Firstly, the reinforcement learning environment for the traffic light control problem is built by defining the three key elements of state, action, and reward. Then, the CNN-based deep Q-network is designed to process the quantized traffic state information to obtain the state-action values. After locally optimizing the deep Q-networks of multiple traffic light agents based on their experience samples, the consensus algorithm is subsequently applied to globally update these agents that are connected over a decentralized communication topology. In this way, the distributed learning agents learn from their neighbors' experience to optimize the modeling process without actually sharing experience data samples. Lastly, homogeneous and heterogeneous traffic flow patterns on different intersections are simulated in SUMO to verify the superiority of the proposed distributed deep Q-networks, with the comparison to the fixed-time strategy, local learning and centralized learning algorithms. The simulation study demonstrates that the distributed learning algorithm without a central server shows comparable performance with centralized learning, which is much better than fixed-time strategy and the local learning method in both homogeneous and heterogeneous traffic scenarios. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control is one of the modern approaches to solve traffic congestion, where reinforcement learning is a widely used method. Conventionally, reinforcement learning is used to determine whether to change the current phase (or choose a traffic phase) after each small interval. One major drawback of these approaches is that it makes the current traffic light phase duration uncertain before the current phase terminates. Directly determining the duration of the traffic light phase can effectively avoid this shortcoming. An adaptive traffic light timing system is proposed in this paper which can directly control the phase duration. In the proposed system, the Q-learning algorithm is employed and the action space is defined as all possible phase durations. In addition, the reward function is redefined to guide the agent to balance more traffic metrics, and the state is redefined to reduce the state space. Finally, the proposed system is evaluated by equal, unequal, and complex traffic scenarios. Results show that the proposed system has a better performance compared with other methods in controlling traffic lights, even on complex traffic scenarios. [ABSTRACT FROM AUTHOR]