Search

This work develops a deep reinforcement learning based approach for Six Degree-of-Freedom (DOF) planetary powered descent and landing. Future Mars missions will require advanced guidance, navigation, and control algorithms for the powered descent phase to target specific surface locations and achieve pinpoint accuracy (landing error ellipse <5 m radius). This requires both a navigation system capable of estimating the lander's state in real-time and a guidance and control system that can map the estimated lander state to a commanded thrust for each lander engine. In this paper, we present a novel integrated guidance and control algorithm designed by applying the principles of reinforcement learning theory. The latter is used to learn a policy mapping the lander's estimated state directly to a commanded thrust for each engine, resulting in accurate and almost fuel-optimal trajectories over a realistic deployment ellipse. Specifically, we use proximal policy optimization, a policy gradient method, to learn the policy. Another contribution of this paper is the use of different discount rates for terminal and shaping rewards, which significantly enhances optimization performance. We present simulation results demonstrating the guidance and control system's performance in a 6-DOF simulation environment and demonstrate robustness to noise and system parameter uncertainty. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel graph-based semi-supervised learning approach for traffic light management in multiple intersections. Specifically, the basic premise behind our paper is that if we know some of the occupied roads and predict which roads will be congested, we can dynamically change traffic lights at the intersections that are connected to the roads anticipated to be congested. Comparative performance evaluations show that the proposed approach can produce comparable average vehicle waiting time and reduce the training/learning time of learning adequate traffic light configurations for all intersections within a few seconds, while a deep learning-based approach can be trained in a few days for learning similar light configurations. • With this work we aimed to optimize traffic signal systems by dynamically setting the system to behave as per predicted traffic flow. • Vehicle flow is predicted using Graph-based semi-supervised and active learning on edge flows. • The systems were created and tested using Simulation Of Urban Mobility (SUMO). • Vehicle data and routes were also generated using this application. • The proposed approach was compared against other methods, such as an occupancy based method, a scoring-based method and reinforcement learning. [ABSTRACT FROM AUTHOR]

Graphical abstract Abstract In this paper, artificial Intelligence (AI) algorithm is used in the Rietveld refinement process instead of the human decision. The program, PowderBot, is developed based on Fullprof engine, which proves the effectiveness of AI in Rietveld refinement. In this program, the decision making in refinement process is modelled as a Markov decision process (MDP), and solved by a reinforcement learning algorithm. PowderBot is designed to be a self-learning system capable of conducting structure refinement without human intervention. The program has already been successfully applied to Rietveld refinements. We hope this paper will encourage more Rietveld programs become more intelligent by the help of AI algorithm. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) is a promising technology to address the resource allocation problem for efficient data transmission in complex network environments. However, most DRL-based resource allocation algorithms suffer from limited feature extraction capabilities and lack scalability and generalization, especially in heterogeneous Industrial Internet of Things (IIoT) environments. In this paper, we develop a lifelong learning architecture that can integrate artificial intelligence (AI) algorithms into the heterogeneous IIoT network for efficient data transmission. Based on this, we propose an intelligent resource allocation algorithm based on dual-attention DRL (DADR) for forwarding node selection and channel access slot allocation in a specific network environment. The proposed DADR algorithm combines the advantages of multi-dimension convolutional attention and multi-head self-attention mechanisms. It can provide local- and global-feature fusion capabilities for distributed nodes while maximizing the performance of data transmission. Furthermore, we present a lifelong federated meta reinforcement learning (LFMRL) that can effectively utilize prior knowledge and enable the DRL agent quickly adapt to a new environment. Specifically, LFMRL adopts a federated meta learning-based knowledge fusion algorithm to fuse the knowledge of learned DADR algorithms and iteratively update the shared model, thereby improving the scalability and generalization of the shared model in heterogeneous IIoT environments. In addition, a simple and efficient knowledge transfer mechanism is enabled to accelerate the DRL model convergence by transferring the knowledge of the shared model to the new environment. Simulation results demonstrate the effectiveness of the proposed algorithms in terms of energy efficiency, data transmission reliability, and network stability. Compared to DADR and FedAvg algorithms, LFMRL algorithm can further reduce the energy consumption, training time, and average forwarding node switching times, while improving packet delivery rate to 99.2%. • A novel lifelong learning architecture for resource allocation in heterogeneous IIoT. • Dual-attention deep reinforcement learning provides local- and global- feature fusion. • Federated meta learning based knowledge fusion learns the commonness of models. • Reinforcement learning based knowledge transfer learns the properties of model. • Knowledge fusion and knowledge transfer provide lifelong learning ability for system. [ABSTRACT FROM AUTHOR]

The past decade has witnessed the rapid development of Artificial Intelligence (AI), especially the explosion of deep learning-related connectionist approaches. This study combines traditional literature review, bibliometric methods, and the Science of Science (SciSci) theory to scrutinize the development context of AI in the last decade on AMiner. 4 4 www.aminer.cn , AMiner is an academic mining system, which will be introduced in the following section. With the assistance of AMiner tools and datasets, this paper aims to describe a further explicit context and evolution of AI in the past decade from the development of connectionist approaches. Five aspects of the past decade are highlighted: self-learning and self-coding algorithms, Recurrent Neural Networks (RNN) algorithms, reinforcement learning, pre-trained models, and other typical deep learning algorithms, which represent the significant progress of this field. By combining these critical parts, we then summarize the current limitations and corresponding future of AI trends in the next decade and discuss some topics about the next generation of AI. Discoveries in this paper will benefit AI research in promoting understanding of the current critical stage and future trends of AI development and the AI industry in the dramatic ascendant for the academic research results transformation and its industrial layout. [ABSTRACT FROM AUTHOR]

Home automation is seen as a potential pillar of the smart city revolution that combines smart mobility, lifestyle and ecosystem governed by intelligent sensors connected to the internet. Households can save money and be more comfortable with automated appliances. The cost of electricity and user comfort are fundamentally contradictory, so they can be presented as a dynamic multi-objective optimization problem with fluctuating priorities for the customer to use various devices at different times. For this reason, this paper proposes an advanced Intelligent Home Energy Management (IHEM) approach based on reinforcement learning to achieve home demand response (DR) efficiency. The optimal formulation of the one-hour-ahead energy consumption scheduling problem is considered a Markov Decision Process (MDP) with discrete time steps. An efficient Neural Network (NN)-based approach with a Q-learning algorithm is developed to address this problem, enabling the IHEM system to achieve better cost-effective scheduling performance. The accurate data of electricity price and energy supplied by the Photovoltaic (PV) system are analyzed in sliding periods by machine learning for uncertainty prediction. Using the newly developed approach, which has the dual objective of minimizing the electricity bill, it is possible to obtain scheduling decisions for appliances and energy storage. The results show that the proposed optimization method reduces the monthly electricity costs by 20% compared to the Integer Linear Programming (ILP)-based HEMS method. • Artificial intelligence (AI) approach to monitor household energy consumption. • An efficient deep reinforcement learning algorithm to control activity recognition in smart homes. • Design and implementation of Intelligent home energy management system. • Converge promptly and considerably reduce operating expenses adopting AI-based approach. [ABSTRACT FROM AUTHOR]

How to design System of Systems has been widely concerned in recent years, especially in military applications. This problem is also known as SoS architecting, which can be boiled down to two subproblems: selecting a number of systems from a set of candidates and specifying the tasks to be completed for each selected system. Essentially, such a problem can be reduced to a combinatorial optimization problem. Traditional exact solvers such as branch-bound algorithm are not efficient enough to deal with large scale cases. Heuristic algorithms are more scalable, but if input changes, these algorithms have to restart the searching process. Re-searching process may take a long time and interfere with the mission achievement of SoS in highly dynamic scenarios, e.g., in the Mosaic Warfare. In this paper, we combine artificial intelligence with SoS architecting and propose a deep reinforcement learning approach DRL-SoSDP for SoS design. Deep neural networks and actor–critic algorithms are used to find the optimal solution with constraints. Evaluation results show that the proposed approach is superior to heuristic algorithms in both solution quality and computation time, especially in large scale cases. DRL-SoSDP can find great solutions in a near real-time manner, showing great potential for cases that require an instant reply. DRL-SoSDP also shows good generalization ability and can find better results than heuristic algorithms even when the scale of SoS is much larger than that in training data. [ABSTRACT FROM AUTHOR]

• Reinforcement learning (RL) formalization of the algorithmic trading problem. • Novel trading strategy based on deep reinforcement learning (DRL), denominated TDQN. • Rigorous performance assessment methodology for algorithmic trading. • TDQN algorithm delivers promising results surpassing benchmark strategies. This scientific research paper presents an innovative approach based on deep reinforcement learning (DRL) to solve the algorithmic trading problem of determining the optimal trading position at any point in time during a trading activity in the stock market. It proposes a novel DRL trading policy so as to maximise the resulting Sharpe ratio performance indicator on a broad range of stock markets. Denominated the Trading Deep Q-Network algorithm (TDQN), this new DRL approach is inspired from the popular DQN algorithm and significantly adapted to the specific algorithmic trading problem at hand. The training of the resulting reinforcement learning (RL) agent is entirely based on the generation of artificial trajectories from a limited set of stock market historical data. In order to objectively assess the performance of trading strategies, the research paper also proposes a novel, more rigorous performance assessment methodology. Following this new performance assessment approach, promising results are reported for the TDQN algorithm. [ABSTRACT FROM AUTHOR]

Improvement of traffic signal control (TSC) efficiency has been found to lead to improved urban transportation and enhanced quality of life. Recently, the use of reinforcement learning (RL) in various areas of TSC has gained significant traction; thus, we conducted a systematic literature review as a systematic, comprehensive, and reproducible review to dissect all the existing research that applied RL in the network-level TSC domain, called as RL in NTSC or RL-NTSC for brevity. The review only targeted the network-level articles that tested the proposed methods in networks with two or more intersections. This review covers 160 peer-reviewed articles from 30 countries published from 1994 to March 2020. The goal of this study is to provide the research community with statistical and conceptual knowledge, summarize existence evidence, characterize RL applications in NTSC domains, explore all applied methods and major first events in the defined scope, and identify areas for further research based on the explored research problems in current research. We analyzed the extracted data from the included articles in the following seven categories: (i) publication and authors' data, (ii) method identification and analysis, (iii) environment attributes and traffic simulation, (iv) application domains of RL-NTSC, (v) major first events of RL-NTSC and authors' key statements, (vi) code availability, and (vii) evaluation. This paper provides a comprehensive view of the past 26 years of research on applying RL to NTSC. It also reveals the role of advancing deep learning methods in the revival of the research area, the rise of using non-commercial microscopic traffic simulators, a lack of interaction between traffic and transportation engineering practitioners and researchers, and a lack of proposal and creation of testbeds which can likely bring different communities together around common goals. • A review on Reinforcement Learning in the network-scale Traffic Signal Control area. • Presents a comprehensive systematic literature review of 160 included articles. • Consolidates and characterizes the existing research on the defined area. • Explores the methods, applications, domains, and first events in the defined scope. • Identifies past and present trends and directions for further research in the area. [ABSTRACT FROM AUTHOR]

The combinatorial optimization (CO) problems on the graph are the core and classic problems in artificial intelligence (AI) and operations research (OR). For example, the Vehicle Routing Problem (VRP) and Traveling Salesman Problem (TSP) are fascinating NP-hard problems and have important significance for the existing transportation system. Traditional methods such as heuristics methods, exact algorithms, and solution solvers can already find approximate solutions on small-scale graphs. However, they are helpless for large-scale graphs and other problems with similar structures. Moreover, traditional methods often require artificially designed heuristic functions to aid decision-making. In recent years, more and more work has focused on applying deep learning and reinforcement learning (RL) to learn heuristics, which allows us to learn the internal structure of the graph end-to-end and find the optimal path under the guidance of heuristic rules. However, most of these still need manual assistance, and the RL method used has the problems of low sampling efficiency and small searchable space. This paper proposes a novel framework (called OmegaZero) based on Alphago Zero, which does not prescribe expert experience or label data but is trained through self-play. We divide the learning into two stages: in the first stage, we employ graph attention network (GAT) and GRU to learn node representations and memory history trajectories. In the second stage, we employ Monte Carlo tree search (MCTS) and deep RL to search for the solution space and train the model. [ABSTRACT FROM AUTHOR]

Artificial intelligence is steadily increasing its impact on everyday life. Therefore, the societal issues of artificial intelligence have become an important concern in the AI research. The presence of data that reflects human biases towards historically discriminated groups defined by sensitive features such as race and gender, results in machine learning models which discriminate against these groups. In order to tackle the impact of bias in data, researchers developed a variety of specialized machine learning algorithms which are able to satisfy different fairness constraints imposed on the model. Group fairness constraints do not fit standard machine learning formulations easily due to their non-differentiable nature. In this paper we developed a technique for learning a fair classifier by Monte Carlo policy gradient method which naturally deals with such non-differentiable constraints. Our methodology focuses on direct optimization of both group fairness metric and predictive performance of the model. In addition, we propose two different variance reduction techniques of gradient estimation. We compare our models to seven other related and state-of-the-art models and demonstrate that they are able to achieve better trade-off between accuracy and unfairness. To the best of our knowledge, this is the first fair classification algorithm which solves the issue of non-differentiable constraints by reinforcement learning techniques. [ABSTRACT FROM AUTHOR]

The Deep Reinforcement Learning (DRL) algorithm, Proximal Policy Optimization (PPO2), is deployed on a custom spacecraft (S/C) build and loss model to determine if an Artificial Intelligence (AI) can learn to monitor satellite constellation health and determine an optimal replacement strategy. A custom environment is created to simulate how S/C are built, launched, generate revenue, and finally decay. The reinforcement learning agent successfully learned an optimal policy for two models: a Simplified Model where the financial cost of actions is ignored; and an Advanced Model where the financial cost of actions is a major element. In both models the AI monitors the constellations and takes multiple strategic and tactical actions to replace satellites to maintain constellation performance. The Simplified Model showed that the PPO2 algorithm was able to converge on an optimal solution after ∼ 200,000 simulations. The Advanced Model was much more difficult for the AI to learn, and thus, the performance drops during the early episodes, but eventually converges to an optimal policy at ∼ 25,000,000 simulations. With the Advanced Model, the AI is taking actions that are successfully providing strategies for constellation management and satellite replacements which include these actions' financial implications. Thus, the methods in this paper provide initial research developments towards a real-world tool and an AI application that can aid various Aerospace businesses in managing Low Earth Orbit (LEO) constellations. This type of AI application may become imperative for deploying and maintaining small satellite mega-constellations. [ABSTRACT FROM AUTHOR]

• The multi-agent defense and attack environment is reconstructed. • Several algorithms are applied to solve the considered problem. • We redefine the state space, the action space and the reward functions accordingly. • Comparison experiments are conducted to show the performance of the employed models. Learning to cooperate among agents has always been an important research topic in artificial intelligence. Multi-agent defense and attack, one of the important issues in multi-agent cooperation, requires multiple agents in the environment to learn effective strategies to achieve their goals. Deep reinforcement learning (DRL) algorithms have natural advantages dealing with continuous control problems especially under situations with dynamic interactions, and have provided new solutions for those long-studied multi-agent cooperation problems. In this paper, we start from deep deterministic policy gradient (DDPG) algorithm and then introduce multi-agent DDPG (MADDPG) to solve the multi-agent defense and attack problem under different situations. We reconstruct the considered environment, redefine the continuous state space, continuous action space, reward functions accordingly, and then apply deep reinforcement learning algorithms to obtain effective decision strategies. Several experiments considering different confrontation scenarios are conducted to validate the feasibility and effectiveness of the DRL-based methods. Experimental results show that through learning the agents can make better decisions, and learning with MADDPG achieves superior performance than learning with other DRL-based models, which also explains the importance and necessity of mastering other agents' information. [ABSTRACT FROM AUTHOR]

• Online network reconfiguration is introduced to manage distribution networks. • A Deep Q-learning algorithm is used to find the optimal network topology. • The proposed algorithm is more scalable than other algorithms. • The computation time of the algorithm is low enough for practical applications. • The algorithm can increase DRES utilization without heavy capital investment. With increasing number of distributed renewable energy sources integrated in power distribution networks, network security issues such as line overloading or bus voltage violations are becoming increasingly common. Traditional capital-intensive system reinforcements could lead to overinvestment. Moreover, active network management solutions, which have emerged as important alternatives, may become a financial burden for distribution system operators or reduce profits for owners of distributed renewable energy sources, or both. To address these limitations, this paper proposes an online network reconfiguration scheme based on a deep reinforcement learning approach. In this scheme, the distribution network operator modifies the network topology to change the power flow when the reliability of network is threatened. Because the variability of distributed renewable energy is large in self-sufficient distribution networks, the reconfiguration process needs to be performed online within short time intervals, which involves the use of conventional algorithms. To solve this problem efficiently, a deep q-learning model is utilized to determine the optimal network topology. Performances of proposed and other algorithms were compared in modified CIGRE 14-bus and IEEE 123-bus test network, as well as varying penalties for frequent switching operation in consideration of physical characteristic of the network. Simulation results demonstrated that the proposed algorithm showed almost identical performances with brute-force search algorithm in both test networks, satisfying network constraints over almost all timespans. Further, the proposed method required very small computation times - under a second per each state and its scalability was verified by comparing the computation time between two test networks. [ABSTRACT FROM AUTHOR]

• DRL-based control methods are proposed for ORC for engine waste heat recovery. • The DRL control for ORC superheat performs considerably better than PID control. • The DRL-based PID control is more robust than the DRL control. • The switching DRL control performs well and exhibits sufficient robustness. • Useful reference and motivation for application of DRL to thermodynamic systems. The organic Rankine cycle (ORC) is a promising technology for engine waste heat recovery. During real-world operation, the engine working condition varies frequently to satisfy the power demand; thus, the transient nature of engine waste heat presents significant control challenges for the ORC. To control the superheat of the ORC precisely under a transient heat source, several optimal control methods have been used such as model predictive control and dynamic programing. However, most of them depend strongly on the accurate prediction of future disturbances. Deep reinforcement learning (DRL) is an artificial-intelligence algorithm that can overcome the aforementioned disadvantage, but the potential of DRL in control of thermodynamic systems has not yet been investigated. Thus, this paper proposes two DRL-based control methods for controlling the superheat of ORC under a transient heat source. One directly uses the DRL agent to learn the control strategy (DRL control), and the other uses the DRL agent to optimize the parameters of the proportional–integral–derivative (PID) controller (DRL-based PID control). Additionally, a switching mechanism between different DRL controllers is proposed for improving the training efficiency and enlarging the operation range of the controller. The results of this study indicate that the DRL agent can satisfactorily perform the control task and optimize the traditional controller under the trained and untrained transient heat source. Specifically, the DRL control can track the reference superheat with an average error of only 0.19 K, whereas that of the traditional PID control is 2.16 K. Furthermore, the proposed switching DRL control exhibits excellent tracking performance with an average error of only 0.21 K and robustness over a wide range of operation conditions. The successful application of DRL demonstrates its considerable potential for the control of thermodynamic systems, providing a useful reference and motivation for the application to other thermodynamic systems. [ABSTRACT FROM AUTHOR]

With advances in smart grid technologies, demand response has played a major role in improving the reliability of grids and reduce the cost for customers. Implementing the demand response scheme for industry is more necessary than for other sectors, because its energy consumption is often considered the largest. This paper proposes a multi-agent deep reinforcement learning based demand response scheme for energy management of discrete manufacturing systems. In this regard, the industrial manufacturing system is initially formulated as a partially-observable Markov game; then, a multi-agent deep deterministic policy gradient algorithm is adopted to obtain the optimal schedule for different machines. A typical lithium-ion battery assembly manufacturing system is used to demonstrate the effectiveness of the proposed scheme. Simulation results show that the presented demand response algorithm can minimize electricity costs and maintain production tasks, as compared to a benchmark without demand response. Moreover, the performance of the multi-agent deep reinforcement learning approach against a mathematical model method is investigated. [ABSTRACT FROM AUTHOR]