Search

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Maintenance planning of networked multi-asset systems is a complex problem due to the inherent individual and collective asset constraints and dynamics as well as the size of the system and interdependencies among assets. Although multi-asset systems have been studied numerous times in the past decades, maintenance planning implications of the system's network characteristics have been barely analysed. Likewise, solutions that consider the network perspective suffer from scalability issues as a network-wide observability is assumed. This paper proposes a network maintenance planning approach based on the decomposition of the multi-asset network into fixed-size localised subnetworks. The overall network maintenance plan is produced by aggregating the subnetwork maintenance plans, which are computed independently via a multi-agent deep reinforcement learning (MARL) algorithm. The results are evaluated against a network-wide approach as well as the commonly-used individual approach. The paper also introduces a systematic approach to integrate the MARL resulting policy in a multi-asset agent-based model. Simulation results of several random asset networks and a large nationwide network infrastructure show that, although a network-wide approach outperforms, on average, other approaches considered, the localised subnetworks approach, provides an acceptable alternative in networks with small-world properties, without the need of a network-wide view. • A Network-wide maintenance approach outperforms individual approaches in networked assets with the topologies analysed. • In analysed networks with small-world properties, the Localised approach shows performance close to the network-wide approach. • A systematic integration of MARL and multi-asset agent-based modelling enables evaluation of maintenance approaches. [ABSTRACT FROM AUTHOR]

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses. [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

This paper addresses a crucial issue in the airline industry by tackling a dynamic pricing problem in the presence of patient customers, a scenario that has gained significance due to the revenue loss of airlines caused by customers' non-myopic decision-making. To effectively capture this non-myopic characteristic, we propose a Markov decision process (MDP) including a history of offered prices as a state variable. In contrast to previous studies, distributions of customers' properties are assumed to be unknown in advance for a more realistic representation of real-world scenarios. To deal with the new challenges of the problem, we propose utilizing a specific learning framework (i.e., deep exploration-based RL) that is unexplored in this domain. The numerical experiments demonstrate that its performance can be improved on the MDP we designed and show that it outperforms the benchmark algorithm. The structures of pricing policies generated from the bootstrapped deep Q-network algorithm imply that airlines should offer high and low prices alternately from the beginning of the sales period rather than increasing prices as time goes on. We also ascertain that more frequent consecutive high-priced periods can increase airlines' revenue in environments with higher customer patience levels. [ABSTRACT FROM AUTHOR]

In recent years, consortium blockchains have attracted considerable interest from the business communities and academia. To satisfy specific application requirements, appropriately configuring or adjusting the parameters is essential but challenging for developers when building the consortium blockchain. In this paper, the parameter configuration and adjustment for the consortium blockchain is transformed as a multi-objective optimization problem, and a novel explainable deep reinforcement learning (DRL) algorithm is proposed to solve the problem. On considering that existing DRL algorithms cannot be directly used in the consortium blockchain as they suffer from lacking of explainability, a causal model for configuring and adjusting the consortium blockchain's parameters is proposed and integrated into the DRL algorithm. The causal model can be used to derive causal explanations of the DRL algorithm to increase its trustworthiness. Furthermore, the causal model-based DRL (C-DRL) algorithm can perform causal inference before taking action to eliminate unreasonable exploration and improve the DRL algorithm's performance. The experimental results demonstrate the proposed algorithm provides the consortium blockchain with adaptive parameter configuration and adjustment to achieve sustainable high performance and security. Furthermore, the proposed algorithm increases the convergence speed of the vanilla algorithm by 49.3% and is more trustworthy than the vanilla DRL algorithm. • To achieve sustainable high performance and security, we transform the parameter configuration and adjustment for the consortium blockchain as a multi-objective optimization problem and provide a formal representation of the problem. • We propose a causal model of consortium blockchain's parameters using the structural causal model (SCM) formalism. • We propose a causal model-based DRL (C-DRL) algorithm to solve the multi-objective optimization problem. The C-DRL algorithm is equipped with the causal model of consortium blockchain's parameters. • The agent can use the causal model to derive the causal explanation for its action to provide trustworthiness. Furthermore, the agent with the causal model can utilize causal inference to eliminate unreasonable exploration and improve the convergence speed of the current DRL algorithm. • Extensive experiments are conducted to validate our proposal. The simulation results show that the agent running the C-DRL algorithm has more reasonable behavior than that running the model-free DRL algorithm. Moreover, the C-DRL algorithm converges faster than the model-free DRL algorithm. [ABSTRACT FROM AUTHOR]

Industrial cyber–physical systems (ICPS) are emerging platforms for various industrial applications. For instance, remote healthcare monitoring, real-time healthcare data generation, and many other applications have been integrated into the ICPS platform. These healthcare applications encompass workflow tasks, such as processing within hospitals, laboratory tests, and insurance companies for patient payments, which necessitate a sequential flow. The external wireless, fog, and cloud services within ICPS face security issues that impact end-users' healthcare applications. Blockchain technology offers an optimal solution for ICPS-enabled applications. However, blockchain technology for the ICPS platform is still vulnerable to cyberattacks, while microservices are essential for executing applications. This paper introduces the novel "Pattern-Proof Malware Validation" (PoPMV) algorithm designed for blockchain in ICPS. It exploits a deep learning model (LSTM) with reinforcement learning techniques to receive feedback and rewards in real-time. The primary objective is to mitigate security vulnerabilities, enhance processing speed, identify both familiar and unfamiliar attacks, and optimize the functionality of ICPS. Simulations demonstrate the superiority of the proposed approach compared to current blockchain frameworks, showcasing dynamic allocation of microservices and improved security with comprehensive attack detection by 30%. [ABSTRACT FROM AUTHOR]

Astronomical seeing is the primary indicator of the observation quality of large-scale optical telescopes. Forecasting astronomical seeing is an essential and challenging task in the operation of large optical telescopes, as it can help observers more efficiently arrange observation tasks as well as providing hints for mechanism optimization of optical telescopes. In this paper, we propose an efficient data-driven astronomical seeing forecasting method by leveraging the environment monitoring data from multiple sensors deployed with optical telescopes. The basic idea is to use a large number of random yet fixed convolution kernels to extract features from the original data, and then train a simple tree-based predictor based on these features. In particular, the raw monitoring data is transformed by differential operations, enabling a more explicit representation of spatial and temporal features; then an elaborate convolution mechanism, including the convolution kernel and the pooling operator, is designed to extract a variety set of features of the input time series, which serve as the foundation for constructing the time series forecasting model. In contrast to existing deep learning models, there is no need to calculate and back-propagate the gradients during the training procedure, making the proposed method much more efficient. We collect real-world monitoring data from Large Sky Area Multi-Object Fiber Spectroscopy Telescope (LAMOST), one of the world's largest optical reflecting telescopes, and conduct extensive empirical evaluations. The experimental results demonstrate that the proposed method can achieve state-of-the-art prediction accuracy, i.e., 0.1125 in MAE, 0.0252 in MSE, and 0.0322 in MAPE, outperforming most deep-learning-based forecasting models. Moreover, it surpasses both machine learning and deep learning approaches in terms of training efficiency with a significant margin. [Display omitted] • The task of forecasting astronomical seeing for optical telescopes is investigated. • A variety of real-world monitoring data of LAMOST is collected. • A new multivariate time series forecasting method — SeeROCKET is proposed. • SeeROCKET achieves very promising performance both in accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Ensemble learning combines several individual models to obtain better generalization performance. Currently, deep learning architectures are showing better performance compared to the shallow or traditional models. Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance. This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. The ensemble models are broadly categorized into bagging, boosting, stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies based deep ensemble models. Applications of deep ensemble models in different domains are also briefly discussed. Finally, we conclude this paper with some potential future research directions. • This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. • The categorizes of deep ensemble models discussed are bagging, boosting and stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies, unsupervised, semi supervised, reinforcement learning and online/incremental, multilabel based deep ensemble models. • Application of deep ensemble models in different domains are discussed. • Finally, this paper provides an outlook towards future research directions. [ABSTRACT FROM AUTHOR]

Pairs trading is a popular classic neutral trading strategy in financial market. Deep reinforcement learning (DRL) has been widely used to improve the performance of this strategy. However, most works primarily focused on setting trading signals, but ignored selecting appropriate trading pairs. In this paper, a novel two-level reinforcement learning framework is proposed, where both pair selection and trading thresholds setting are involved. For pair selection, an Extended Option-Critic (EOC) method is utilized, which allows the agent to select trading pair on non-fixed length of time intervals. For trading thresholds setting, a three-agent Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is used for setting the opening and stop-loss thresholds as well as decide whether to trade. The simulation results in the Chinese futures market demonstrate that our proposed method achieves higher returns compared to traditional methods and popular reinforcement learning approaches. • A novel two-level reinforcement learning framework for future trading is proposed. • Extended Option-Critic method for non-fixed length of time intervals. • MADDPG method with the cooperation of three agents for trading. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning systems are now a hot topic in Machine Learning for their effectiveness in many complex tasks, but their application in safety-critical domains (e.g., robot control or self-autonomous driving) remains dangerous without mechanism to detect and prevent risk situations. In Deep RL, such risk is mostly in the form of adversarial attacks, which introduce small perturbations to sensor inputs with the aim of changing the network-based decisions and thus cause catastrophic situations. In the light of these dangers, a promising line of research is that of providing these Deep RL algorithms with suitable defenses, especially when deploying in real environments. This paper suggests that this line of research could be greatly improved by the concepts from the existing research field of Safe Reinforcement Learning, which has been postulated as a family of RL algorithms capable of providing defenses against many forms of risks. However, the connections between Safe RL and the design of defenses against adversarial attacks in Deep RL remain largely unexplored. This paper seeks to explore precisely some of these connections. In particular, this paper proposes to reuse some of the concepts from existing Safe RL algorithms to create a novel and effective instance-based defense for the deployment stage of Deep RL policies. The proposed algorithm uses a risk function based on how far a state is from the state space known by the agent, that allows identifying and preventing adversarial situations. The success of the proposed defense has been evaluated in 4 Atari games. [ABSTRACT FROM AUTHOR]

In recent research, problems with biased datasets or domain shift have presented challenges to the practical applications of deep learning methods. In this paper, we propose a simple method using adversarial learning combined with contrastive learning and domain adaptation to solve the domain-shift problem. Because domain shift is caused by differences in the distribution of data across domains, different approaches have been proposed to resolve this by rendering the data distributions closer through adversarial training. Contrastive learning is a method of acquiring valid feature representations for downstream tasks. By combining these approaches, we aim to achieve better domain adaptation. The proposed method is simple and intuitive. By introducing a domain discriminator into SimCLR, which is a typical contrastive learning model, and training it in an adversarial manner, the feature vectors of the source and target domains are rendered closer to acquire domain-invariant features. The proposed approach facilitates high-performance pretraining without labels and demonstrates a significant improvement in accuracy in comparison to standard benchmark methods, including conventional supervised models and SimCLR. • We investigated an unsupervised domain adaptation task. • Combining adversarial and contrastive learning can address the domain-shift issue. • The proposed method is practical and implementation is very simple. [ABSTRACT FROM AUTHOR]

Home automation is seen as a potential pillar of the smart city revolution that combines smart mobility, lifestyle and ecosystem governed by intelligent sensors connected to the internet. Households can save money and be more comfortable with automated appliances. The cost of electricity and user comfort are fundamentally contradictory, so they can be presented as a dynamic multi-objective optimization problem with fluctuating priorities for the customer to use various devices at different times. For this reason, this paper proposes an advanced Intelligent Home Energy Management (IHEM) approach based on reinforcement learning to achieve home demand response (DR) efficiency. The optimal formulation of the one-hour-ahead energy consumption scheduling problem is considered a Markov Decision Process (MDP) with discrete time steps. An efficient Neural Network (NN)-based approach with a Q-learning algorithm is developed to address this problem, enabling the IHEM system to achieve better cost-effective scheduling performance. The accurate data of electricity price and energy supplied by the Photovoltaic (PV) system are analyzed in sliding periods by machine learning for uncertainty prediction. Using the newly developed approach, which has the dual objective of minimizing the electricity bill, it is possible to obtain scheduling decisions for appliances and energy storage. The results show that the proposed optimization method reduces the monthly electricity costs by 20% compared to the Integer Linear Programming (ILP)-based HEMS method. • Artificial intelligence (AI) approach to monitor household energy consumption. • An efficient deep reinforcement learning algorithm to control activity recognition in smart homes. • Design and implementation of Intelligent home energy management system. • Converge promptly and considerably reduce operating expenses adopting AI-based approach. [ABSTRACT FROM AUTHOR]

Future industrial automation systems are anticipated to be shaped by intelligent technologies that allow for the adaptability of machines to the variations and uncertainties in processes and work environments. This paper is motivated by the need for devising new intelligent methods that enable efficient and scalable training of collaborative robots on a variety of tasks that foster their adaptability to new tasks and environments. Recent advances in deep Reinforcement Learning (RL) provide new possibilities to realize this vision. The state-of-the-art in deep RL offers proven algorithms that enable autonomous learning and mastery of a variety of robotic manipulation tasks with minimal human intervention. However, current deep RL algorithms predominantly specialize in a narrow range of tasks, are sample inefficient, and lack sufficient stability, which hinders their adoption in real-life, industrial settings. This paper develops and tests a Hyper-Actor Soft Actor–Critic (HASAC) deep RL framework based on the notions of task modularization and transfer learning to tackle this limitation. The goal of the proposed HASAC is to enhance an agent's adaptability to new tasks by transferring the learned policies of former tasks to the new task through a "hyper-actor". The HASAC framework is tested on the virtual robotic manipulation benchmark, Meta-World. Numerical experiments indicate superior performance by HASAC over state-of-the-art deep RL algorithms in terms of reward value, success rate, and task completion time. [ABSTRACT FROM AUTHOR]

Missions involving Unmanned Aerial Vehicle usually consist of reaching a set of regions, performing some actions in each region, and returning to a determined depot after all the regions have been successfully visited or before the fuel/battery is totally consumed. Hence, planning a route becomes an important task for many applications, especially if a team of Unmanned Aerial Vehicles is considered. From this team, coordination and cooperation are expected to optimize results of the mission. In this paper, a system for managing multiple cooperative Unmanned Aerial Vehicles is presented. This system divides the routing problem into two stages: initial planning and routing solving. Initial planning is a first step where the regions to be visited are grouped in multiple clusters according to a distance criterion, with each cluster being assigned to each of the Unmanned Aerial Vehicles. Routing solving computes the best route for every agent considering the clusters of the initial planning and a variant of the Orienteering Problem. This variant introduces the concept of shared regions, allowing an Unmanned Aerial Vehicle to visit regions from other clusters and compensating for the suboptimal region clustering of the previous stage. The Orienteering Problem with shared regions is solved using the deep learning architecture Transformer along with a deep reinforcement learning framework. This architecture is able to obtain high-quality solutions much faster than conventional optimization approaches. Extensive results and comparison with other Combinatorial Optimization algorithms, including cooperative and non-cooperative scenarios, have been performed to show the benefits of the proposed solution. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning has shown promising results in learning policies for complex sequential decision-making tasks. However, different adversarial attack strategies have revealed the weakness of these policies to perturbations to their observations. Most of these attacks have been built on existing adversarial example crafting techniques used to fool classifiers, where an adversarial attack is considered a success if it makes the classifier outputs any wrong class. The major drawback of these approaches when applied to decision-making tasks is that they are blind for long-term goals. In contrast, this paper suggests that it is more appropriate to view the attack process as a sequential optimization problem, with the aim of learning a sequence of attacks, where the attacker must consider the long-term effects of each attack. In this paper, we propose that such an attack policy must be learned with two objectives in view. On the one hand, the attack must pursue the maximum performance loss of the attacked policy. On the other hand, it also should minimize the cost of the attacks. Therefore, in this paper we propose a novel modelization of the process of learning an attack policy as a Multi-objective Markov Decision Process with two objectives: maximizing the performance loss of the attacked policy and minimizing the cost of the attacks. We also reveal the conflicting nature of these two objectives and use a Multi-objective Reinforcement Learning algorithm to draw the Pareto fronts for four well-known tasks: the GridWorld, the Cartpole, the Mountain car and the Breakout. [ABSTRACT FROM AUTHOR]

Reinforcement Learning (RL) has been employed in many applications of robotics and has steadily been gaining traction in the field of Autonomous Driving (AD). This paper proposes a Deep Reinforcement Learning based approach for lateral Vehicle Motion Control (VMC), and explores the generalization capabilities of the approach. The proposed methodology uses a sequence of waypoints generated from a planning module of an AD stack as the input. The network has been trained to predict accurate steering commands to follow the given trajectory. In this paper we detail our implementation and share our learning experience on real-vehicle deployment of the RL based controller. Our experiments yield promising results with an agent trained on less than 4 h of simulated driving experience without any real-world data. The trained agent is able to successfully complete unseen and more complex tracks using different unseen vehicle models. The agent safely reached up to 150 km/h in simulation and up to 60 km/h in a real-life Sport Utility Vehicle (SUV) weighing more than 2000 kg. [ABSTRACT FROM AUTHOR]

This paper proposes a Deep Reinforcement Learning (DRL) algorithm that expands the Random Neural Network (RNN) Reinforcement Learning (RL) method to include the previous learnings entirely from previous rewards, rather than only the actual one. The Random Neural Network weighs are updated with the current reward and the previous values, including time and memory. This addition makes DRL slower to make decisions, although it also increases its performance in some experiments. Several configurations to introduce DRL, such as sampling rate and memory duration, are also proposed and analysed in this article. The proposed DRL algorithm is included in a decision process that predicts trends: upward, downward and equal market directions in addition to values. Experimental results based on market prices demonstrate that the addition of Deep Learning to the Reinforcement Learning algorithm increases its performance slightly in some experiments; however, it also increases its computational cost. In random environments such as the stock market, it is preferable to make decisions based on the previous values (short memory) rather than historical records (long memory). [ABSTRACT FROM AUTHOR]

When a complex large power system is in an emergency, the conventional discrete emergency control strategy implemented will cause excess load or derivative accidents like line power overload, thereby raising the operation risk. To overcome the problems of excessive/insufficient regulation and subsequent accidents in the above context, this paper proposes a wide area automatic generation control (WA-AGC) framework, which integrates the emergency control strategy and a performance-based frequency regulation market mechanism. According to the frequency status of the power system, WA-AGC divides the AGC into four intervals, emergency AGC (EAGC), conventional AGC (CAGC), AGC transition and optimal power flow (OPF). These four together realize a comprehensive optimization of frequency and system stability as well as economy Based on the above framework, a swarm agent exploration distributed multiple delayed deep policy gradient algorithm (SAE-MD3) is developed, which uses multiple explorers with different exploration strategies for distributed optimization. In addition, several technologies are introduced to prevent Q value overestimation and generate a more robust optimal AGC strategy. Afterward, the effectiveness and feasibility of WA-AGC are verified through the simulations of an IEEE-9 two-area system and an IEEE-118 two-area system. Compared to conventional AGC strategies, the WA-AGC algorithm reduces the constraint violation time of the power line by 92.06% and the power generation cost by 0.27% as well as improves the CPS1 index by 0.04%. • A novel AGC framework using WAMS data is proposed to substitute the-conventional AGC. • The WA-AGC framework can achieve the optimal dispatch for different objective. • A novel deep reinforcement learning algorithm with better robustness is introduced. • The WA-AGC solves the excess/lack of regulation problem and derivative accidents. • The WA-AGC considers the impact of emergency fault on the power grid. [ABSTRACT FROM AUTHOR]

The control performance of conventional analytic algorithms for double-fed induction generator-based wind turbines is a fixed feature, which needs to be optimized by optimization processes. To avoid the optimization processes and update control strategies online, this paper proposes an online control algorithm based on the quantum process, deep belief networks, and reinforcement learning for double-fed induction generator-based wind turbines. The proposed approach is named quantum deep reinforcement learning, consisting of deep belief networks, one reinforcement learning framework, and multiple subsidiary reinforcement learning parts with quantum processes. The quantum deep reinforcement learning can update the control strategy online with general initialization for dynamic systems, avoid optimal local solutions, and predict the next systemic states of double-fed induction generator-based wind turbines. The proportional–integral–derivative, fractional-order proportional–integral–derivative, active disturbance rejection controller, reinforcement learning, and the quantum deep reinforcement learning are compared under four cases, i.e., variable wind speed with step magnitude, variable wind speed with sine magnitude, voltage dropout of power grids, and both variable wind speed with random magnitude and voltage dropout of power grids. Consequently, the proposed quantum deep reinforcement learning can obtain better control performance for double-fed induction generator-based wind turbines with smaller integrated absolute error, integral squared error, and integrated time-weighted absolute error of the control error than other compared methods. • Parameters of conventional control algorithms need to be optimized offline. • An online control algorithm is proposed to avoid the optimization processes. • Quantum process and deep reinforcement learning are combined for the online method. • The method can avoid local optimal solutions and can predict next systemic states. • Higher control performance is effectively obtained for controllers of wind turbines. [ABSTRACT FROM AUTHOR]

The combinatorial optimization (CO) problems on the graph are the core and classic problems in artificial intelligence (AI) and operations research (OR). For example, the Vehicle Routing Problem (VRP) and Traveling Salesman Problem (TSP) are fascinating NP-hard problems and have important significance for the existing transportation system. Traditional methods such as heuristics methods, exact algorithms, and solution solvers can already find approximate solutions on small-scale graphs. However, they are helpless for large-scale graphs and other problems with similar structures. Moreover, traditional methods often require artificially designed heuristic functions to aid decision-making. In recent years, more and more work has focused on applying deep learning and reinforcement learning (RL) to learn heuristics, which allows us to learn the internal structure of the graph end-to-end and find the optimal path under the guidance of heuristic rules. However, most of these still need manual assistance, and the RL method used has the problems of low sampling efficiency and small searchable space. This paper proposes a novel framework (called OmegaZero) based on Alphago Zero, which does not prescribe expert experience or label data but is trained through self-play. We divide the learning into two stages: in the first stage, we employ graph attention network (GAT) and GRU to learn node representations and memory history trajectories. In the second stage, we employ Monte Carlo tree search (MCTS) and deep RL to search for the solution space and train the model. [ABSTRACT FROM AUTHOR]

Artificial intelligence is steadily increasing its impact on everyday life. Therefore, the societal issues of artificial intelligence have become an important concern in the AI research. The presence of data that reflects human biases towards historically discriminated groups defined by sensitive features such as race and gender, results in machine learning models which discriminate against these groups. In order to tackle the impact of bias in data, researchers developed a variety of specialized machine learning algorithms which are able to satisfy different fairness constraints imposed on the model. Group fairness constraints do not fit standard machine learning formulations easily due to their non-differentiable nature. In this paper we developed a technique for learning a fair classifier by Monte Carlo policy gradient method which naturally deals with such non-differentiable constraints. Our methodology focuses on direct optimization of both group fairness metric and predictive performance of the model. In addition, we propose two different variance reduction techniques of gradient estimation. We compare our models to seven other related and state-of-the-art models and demonstrate that they are able to achieve better trade-off between accuracy and unfairness. To the best of our knowledge, this is the first fair classification algorithm which solves the issue of non-differentiable constraints by reinforcement learning techniques. [ABSTRACT FROM AUTHOR]

Autonomous mobile robots employed in industrial applications often operate in complex and uncertain environments. In this paper we propose an approach based on an extension of Partially Observable Monte Carlo Planning (POMCP) for robot velocity regulation in industrial-like environments characterized by uncertain motion difficulties. The velocity selected by POMCP is used by a standard engine controller which deals with path planning. This two-layer approach allows POMCP to exploit prior knowledge on the relationships between task similarities to improve performance in terms of time spent to traverse a path with obstacles. We also propose three measures to support human-understanding of the strategy used by POMCP to improve the performance. The overall architecture is tested on a Turtlebot3 in two environments, a rectangular path and a realistic production line in a research lab. Tests performed on a C++ simulator confirm the capability of the proposed approach to profitably use prior knowledge, achieving a performance improvement from 0.7% to 3.1% depending on the complexity of the path. Experiments on a Unity simulator show that the proposed two-layer approach outperforms also single-layer approaches based only on the engine controller (i.e., without the POMCP layer). In this case the performance improvement is up to 37% comparing to a state-of-the-art deep reinforcement learning engine controller, and up to 51% comparing to the standard ROS engine controller. Finally, experiments in a real-world testing arena confirm the possibility to run the approach on real robots. [ABSTRACT FROM AUTHOR]

The Deep Reinforcement Learning (DRL) algorithm, Proximal Policy Optimization (PPO2), is deployed on a custom spacecraft (S/C) build and loss model to determine if an Artificial Intelligence (AI) can learn to monitor satellite constellation health and determine an optimal replacement strategy. A custom environment is created to simulate how S/C are built, launched, generate revenue, and finally decay. The reinforcement learning agent successfully learned an optimal policy for two models: a Simplified Model where the financial cost of actions is ignored; and an Advanced Model where the financial cost of actions is a major element. In both models the AI monitors the constellations and takes multiple strategic and tactical actions to replace satellites to maintain constellation performance. The Simplified Model showed that the PPO2 algorithm was able to converge on an optimal solution after ∼ 200,000 simulations. The Advanced Model was much more difficult for the AI to learn, and thus, the performance drops during the early episodes, but eventually converges to an optimal policy at ∼ 25,000,000 simulations. With the Advanced Model, the AI is taking actions that are successfully providing strategies for constellation management and satellite replacements which include these actions' financial implications. Thus, the methods in this paper provide initial research developments towards a real-world tool and an AI application that can aid various Aerospace businesses in managing Low Earth Orbit (LEO) constellations. This type of AI application may become imperative for deploying and maintaining small satellite mega-constellations. [ABSTRACT FROM AUTHOR]

Quality of Service (QoS) provisioning is based on various network management techniques including resource management and medium access control (MAC). Various techniques have been introduced to automate networking decisions, particularly at the MAC layer. Deep reinforcement learning (DRL), as a solution to sequential decision making problems, is a combination of the power of deep learning (DL), to represent and comprehend the world, with reinforcement learning (RL), to understand the environment and act rationally. In this paper, we present a survey on the applications of DRL in QoS provisioning at the MAC layer. First, we present the basic concepts of QoS and DRL. Second, we classify the main challenges in the context of QoS provisioning at the MAC layer, including medium access and data rate control, and resource sharing and scheduling. Third, we review various DRL algorithms employed to support QoS at the MAC layer, by analyzing, comparing, and identifying their pros and cons. Furthermore, we outline a number of important open research problems and suggest some avenues for future research. [ABSTRACT FROM AUTHOR]

The intelligent control of the traffic signal is critical to the optimization of transportation systems. To achieve global optimal traffic efficiency in large-scale road networks, recent works have focused on coordination among intersections, which have shown promising results. However, existing studies paid more attention to observations sharing among intersections (both explicit and implicit) and did not care about the consequences after decisions. In this paper, we design a multi-agent coordination framework based on Deep Reinforcement Learning method for traffic signal control, defined as γ - Reward that includes both original γ - Reward and γ - Attention-Reward. Specifically, we propose the Spatial Differentiation method for coordination which uses the temporal–spatial information in the replay buffer to amend the reward of each action. A concise theoretical analysis that proves the proposed model can converge to Nash equilibrium is given. By extending the idea of Markov Chain to the dimension of space–time, this truly decentralized coordination mechanism replaces the graph attention method and realizes the decoupling of the road network, which is more scalable and more in line with practice. The simulation results show that the proposed model remains a state-of-the-art performance even not use a centralized setting. Code is available in https://github.com/Skylark0924/Gamma_Reward. [ABSTRACT FROM AUTHOR]

This paper introduces a new scalable multi-objective deep reinforcement learning (MODRL) framework based on deep Q-networks. We develop a high-performance MODRL framework that supports both single-policy and multi-policy strategies, as well as both linear and non-linear approaches to action selection. The experimental results on two benchmark problems (two-objective deep sea treasure environment and three-objective Mountain Car problem) indicate that the proposed framework is able to find the Pareto-optimal solutions effectively. The proposed framework is generic and highly modularized, which allows the integration of different deep reinforcement learning algorithms in different complex problem domains. This therefore overcomes many disadvantages involved with standard multi-objective reinforcement learning methods in the current literature. The proposed framework acts as a testbed platform that accelerates the development of MODRL for solving increasingly complicated multi-objective problems. [ABSTRACT FROM AUTHOR]

Deep learning applications have been thriving over the last decade in many different domains, including computer vision and natural language understanding. The drivers for the vibrant development of deep learning have been the availability of abundant data, breakthroughs of algorithms and the advancements in hardware. Despite the fact that complex industrial assets have been extensively monitored and large amounts of condition monitoring signals have been collected, the application of deep learning approaches for detecting, diagnosing and predicting faults of complex industrial assets has been limited. The current paper provides a thorough evaluation of the current developments, drivers, challenges, potential solutions and future research needs in the field of deep learning applied to Prognostics and Health Management (PHM) applications. [ABSTRACT FROM AUTHOR]