Search

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Maintenance planning of networked multi-asset systems is a complex problem due to the inherent individual and collective asset constraints and dynamics as well as the size of the system and interdependencies among assets. Although multi-asset systems have been studied numerous times in the past decades, maintenance planning implications of the system's network characteristics have been barely analysed. Likewise, solutions that consider the network perspective suffer from scalability issues as a network-wide observability is assumed. This paper proposes a network maintenance planning approach based on the decomposition of the multi-asset network into fixed-size localised subnetworks. The overall network maintenance plan is produced by aggregating the subnetwork maintenance plans, which are computed independently via a multi-agent deep reinforcement learning (MARL) algorithm. The results are evaluated against a network-wide approach as well as the commonly-used individual approach. The paper also introduces a systematic approach to integrate the MARL resulting policy in a multi-asset agent-based model. Simulation results of several random asset networks and a large nationwide network infrastructure show that, although a network-wide approach outperforms, on average, other approaches considered, the localised subnetworks approach, provides an acceptable alternative in networks with small-world properties, without the need of a network-wide view. • A Network-wide maintenance approach outperforms individual approaches in networked assets with the topologies analysed. • In analysed networks with small-world properties, the Localised approach shows performance close to the network-wide approach. • A systematic integration of MARL and multi-asset agent-based modelling enables evaluation of maintenance approaches. [ABSTRACT FROM AUTHOR]

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses. [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

This paper addresses a crucial issue in the airline industry by tackling a dynamic pricing problem in the presence of patient customers, a scenario that has gained significance due to the revenue loss of airlines caused by customers' non-myopic decision-making. To effectively capture this non-myopic characteristic, we propose a Markov decision process (MDP) including a history of offered prices as a state variable. In contrast to previous studies, distributions of customers' properties are assumed to be unknown in advance for a more realistic representation of real-world scenarios. To deal with the new challenges of the problem, we propose utilizing a specific learning framework (i.e., deep exploration-based RL) that is unexplored in this domain. The numerical experiments demonstrate that its performance can be improved on the MDP we designed and show that it outperforms the benchmark algorithm. The structures of pricing policies generated from the bootstrapped deep Q-network algorithm imply that airlines should offer high and low prices alternately from the beginning of the sales period rather than increasing prices as time goes on. We also ascertain that more frequent consecutive high-priced periods can increase airlines' revenue in environments with higher customer patience levels. [ABSTRACT FROM AUTHOR]

In recent years, consortium blockchains have attracted considerable interest from the business communities and academia. To satisfy specific application requirements, appropriately configuring or adjusting the parameters is essential but challenging for developers when building the consortium blockchain. In this paper, the parameter configuration and adjustment for the consortium blockchain is transformed as a multi-objective optimization problem, and a novel explainable deep reinforcement learning (DRL) algorithm is proposed to solve the problem. On considering that existing DRL algorithms cannot be directly used in the consortium blockchain as they suffer from lacking of explainability, a causal model for configuring and adjusting the consortium blockchain's parameters is proposed and integrated into the DRL algorithm. The causal model can be used to derive causal explanations of the DRL algorithm to increase its trustworthiness. Furthermore, the causal model-based DRL (C-DRL) algorithm can perform causal inference before taking action to eliminate unreasonable exploration and improve the DRL algorithm's performance. The experimental results demonstrate the proposed algorithm provides the consortium blockchain with adaptive parameter configuration and adjustment to achieve sustainable high performance and security. Furthermore, the proposed algorithm increases the convergence speed of the vanilla algorithm by 49.3% and is more trustworthy than the vanilla DRL algorithm. • To achieve sustainable high performance and security, we transform the parameter configuration and adjustment for the consortium blockchain as a multi-objective optimization problem and provide a formal representation of the problem. • We propose a causal model of consortium blockchain's parameters using the structural causal model (SCM) formalism. • We propose a causal model-based DRL (C-DRL) algorithm to solve the multi-objective optimization problem. The C-DRL algorithm is equipped with the causal model of consortium blockchain's parameters. • The agent can use the causal model to derive the causal explanation for its action to provide trustworthiness. Furthermore, the agent with the causal model can utilize causal inference to eliminate unreasonable exploration and improve the convergence speed of the current DRL algorithm. • Extensive experiments are conducted to validate our proposal. The simulation results show that the agent running the C-DRL algorithm has more reasonable behavior than that running the model-free DRL algorithm. Moreover, the C-DRL algorithm converges faster than the model-free DRL algorithm. [ABSTRACT FROM AUTHOR]

Pairs trading is a popular classic neutral trading strategy in financial market. Deep reinforcement learning (DRL) has been widely used to improve the performance of this strategy. However, most works primarily focused on setting trading signals, but ignored selecting appropriate trading pairs. In this paper, a novel two-level reinforcement learning framework is proposed, where both pair selection and trading thresholds setting are involved. For pair selection, an Extended Option-Critic (EOC) method is utilized, which allows the agent to select trading pair on non-fixed length of time intervals. For trading thresholds setting, a three-agent Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is used for setting the opening and stop-loss thresholds as well as decide whether to trade. The simulation results in the Chinese futures market demonstrate that our proposed method achieves higher returns compared to traditional methods and popular reinforcement learning approaches. • A novel two-level reinforcement learning framework for future trading is proposed. • Extended Option-Critic method for non-fixed length of time intervals. • MADDPG method with the cooperation of three agents for trading. [ABSTRACT FROM AUTHOR]

Future industrial automation systems are anticipated to be shaped by intelligent technologies that allow for the adaptability of machines to the variations and uncertainties in processes and work environments. This paper is motivated by the need for devising new intelligent methods that enable efficient and scalable training of collaborative robots on a variety of tasks that foster their adaptability to new tasks and environments. Recent advances in deep Reinforcement Learning (RL) provide new possibilities to realize this vision. The state-of-the-art in deep RL offers proven algorithms that enable autonomous learning and mastery of a variety of robotic manipulation tasks with minimal human intervention. However, current deep RL algorithms predominantly specialize in a narrow range of tasks, are sample inefficient, and lack sufficient stability, which hinders their adoption in real-life, industrial settings. This paper develops and tests a Hyper-Actor Soft Actor–Critic (HASAC) deep RL framework based on the notions of task modularization and transfer learning to tackle this limitation. The goal of the proposed HASAC is to enhance an agent's adaptability to new tasks by transferring the learned policies of former tasks to the new task through a "hyper-actor". The HASAC framework is tested on the virtual robotic manipulation benchmark, Meta-World. Numerical experiments indicate superior performance by HASAC over state-of-the-art deep RL algorithms in terms of reward value, success rate, and task completion time. [ABSTRACT FROM AUTHOR]

Missions involving Unmanned Aerial Vehicle usually consist of reaching a set of regions, performing some actions in each region, and returning to a determined depot after all the regions have been successfully visited or before the fuel/battery is totally consumed. Hence, planning a route becomes an important task for many applications, especially if a team of Unmanned Aerial Vehicles is considered. From this team, coordination and cooperation are expected to optimize results of the mission. In this paper, a system for managing multiple cooperative Unmanned Aerial Vehicles is presented. This system divides the routing problem into two stages: initial planning and routing solving. Initial planning is a first step where the regions to be visited are grouped in multiple clusters according to a distance criterion, with each cluster being assigned to each of the Unmanned Aerial Vehicles. Routing solving computes the best route for every agent considering the clusters of the initial planning and a variant of the Orienteering Problem. This variant introduces the concept of shared regions, allowing an Unmanned Aerial Vehicle to visit regions from other clusters and compensating for the suboptimal region clustering of the previous stage. The Orienteering Problem with shared regions is solved using the deep learning architecture Transformer along with a deep reinforcement learning framework. This architecture is able to obtain high-quality solutions much faster than conventional optimization approaches. Extensive results and comparison with other Combinatorial Optimization algorithms, including cooperative and non-cooperative scenarios, have been performed to show the benefits of the proposed solution. [ABSTRACT FROM AUTHOR]

This paper proposes a Deep Reinforcement Learning (DRL) algorithm that expands the Random Neural Network (RNN) Reinforcement Learning (RL) method to include the previous learnings entirely from previous rewards, rather than only the actual one. The Random Neural Network weighs are updated with the current reward and the previous values, including time and memory. This addition makes DRL slower to make decisions, although it also increases its performance in some experiments. Several configurations to introduce DRL, such as sampling rate and memory duration, are also proposed and analysed in this article. The proposed DRL algorithm is included in a decision process that predicts trends: upward, downward and equal market directions in addition to values. Experimental results based on market prices demonstrate that the addition of Deep Learning to the Reinforcement Learning algorithm increases its performance slightly in some experiments; however, it also increases its computational cost. In random environments such as the stock market, it is preferable to make decisions based on the previous values (short memory) rather than historical records (long memory). [ABSTRACT FROM AUTHOR]

Quality of Service (QoS) provisioning is based on various network management techniques including resource management and medium access control (MAC). Various techniques have been introduced to automate networking decisions, particularly at the MAC layer. Deep reinforcement learning (DRL), as a solution to sequential decision making problems, is a combination of the power of deep learning (DL), to represent and comprehend the world, with reinforcement learning (RL), to understand the environment and act rationally. In this paper, we present a survey on the applications of DRL in QoS provisioning at the MAC layer. First, we present the basic concepts of QoS and DRL. Second, we classify the main challenges in the context of QoS provisioning at the MAC layer, including medium access and data rate control, and resource sharing and scheduling. Third, we review various DRL algorithms employed to support QoS at the MAC layer, by analyzing, comparing, and identifying their pros and cons. Furthermore, we outline a number of important open research problems and suggest some avenues for future research. [ABSTRACT FROM AUTHOR]

The intelligent control of the traffic signal is critical to the optimization of transportation systems. To achieve global optimal traffic efficiency in large-scale road networks, recent works have focused on coordination among intersections, which have shown promising results. However, existing studies paid more attention to observations sharing among intersections (both explicit and implicit) and did not care about the consequences after decisions. In this paper, we design a multi-agent coordination framework based on Deep Reinforcement Learning method for traffic signal control, defined as γ - Reward that includes both original γ - Reward and γ - Attention-Reward. Specifically, we propose the Spatial Differentiation method for coordination which uses the temporal–spatial information in the replay buffer to amend the reward of each action. A concise theoretical analysis that proves the proposed model can converge to Nash equilibrium is given. By extending the idea of Markov Chain to the dimension of space–time, this truly decentralized coordination mechanism replaces the graph attention method and realizes the decoupling of the road network, which is more scalable and more in line with practice. The simulation results show that the proposed model remains a state-of-the-art performance even not use a centralized setting. Code is available in https://github.com/Skylark0924/Gamma_Reward. [ABSTRACT FROM AUTHOR]

Deep learning applications have been thriving over the last decade in many different domains, including computer vision and natural language understanding. The drivers for the vibrant development of deep learning have been the availability of abundant data, breakthroughs of algorithms and the advancements in hardware. Despite the fact that complex industrial assets have been extensively monitored and large amounts of condition monitoring signals have been collected, the application of deep learning approaches for detecting, diagnosing and predicting faults of complex industrial assets has been limited. The current paper provides a thorough evaluation of the current developments, drivers, challenges, potential solutions and future research needs in the field of deep learning applied to Prognostics and Health Management (PHM) applications. [ABSTRACT FROM AUTHOR]