Search

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) can be described as a multi-agent cooperative game. To realize cooperation, multi-agent reinforcement learning (MARL) is a significant approach, with communication being a core component. The large-scale traffic signals and the partially observable information in TSC pose a considerable challenge in finding the optimal joint control policy. This paper proposed a deep MARL model named attentional graph relations communications network (AGRCNet). Based on the Actor-Critic framework, AGRCNet designs a communication network to exchange observation information with agents to help obtain the optimal joint action, reducing the decision error caused by the partially observable condition. Specifically, through the communication network, the chain propagation of graph attention networks (GAT) and graph convolutional networks is used to expand the receptive domain of agents, improve communication efficiency and promote cooperative behavior. We simulate the traffic situation near the Nanjing Yangtze River Bridge in Simulation of Urban MObility. With a compound reward, our method performs best. Meanwhile, AGRCNet is applied to two abstract environments, and the results show that our approach can also adapt to dynamic agent relationships and is more efficient than comparison algorithms. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration. [ABSTRACT FROM AUTHOR]

The paper is devoted to scheduling in multiagent systems in the framework of the Flatland 3 competition. The main aim of this competition is to develop an algorithm for the effective control of dense traffic in complex railroad networks according to a given schedule. The proposed solution is based on reinforcement learning. To adapt this method to the particular scheduling problem, a novel approach based on structuring the reward function that stimulates an agent to adhere to its schedule was developed. The architecture of the proposed model is based on a multiagent version of centralized critic with proximal policy optimization (PPO) learning. In addition, a curriculum learning strategy was developed and implemented. This allowed the agent to cope with each level of complexity on time and train the model in more difficult conditions. The proposed solution won first place in the Flatland 3 competition in the reinforcement learning track. [ABSTRACT FROM AUTHOR]

Traffic lights are an important controlling factor in traffic flows, and good policies will facilitate traffic congestion. A car's waiting time is highly related to the period in which the traffic lights are green or red; thus, a proper controlling policy reduces the waiting time and speeds up the car movement. Emergency vehicles always have a higher priority than other cars and need to reach their destination faster. In the last decade, a lot of research has been done to solve the traffic light scheduling issue using artificial intelligence techniques. Reinforcement learning with simulating human learning has the outstanding performance to solve complex problems, such as traffic lights control. In this study, a deep reinforcement learning traffic light control was implemented at a crossroad using real-time traffic information with an emphasis on emergency vehicles. The agent learns to adjust its policies to prioritize the emergency cars over the other cars. The results show less waiting time and speed up the cars' movement. It also indicates the emergency vehicles will cross the intersection with the least delay. With 200 to 600 cars and one emergency vehicle, the average delay decreases 2–16% in total using reinforcement learning. Considering more emergency vehicles, there is a 27–40% decrease in average delay for emergency vehicles. [ABSTRACT FROM AUTHOR]