Search

The development of quantum machine learning (QML) has received a lot of interest recently thanks to developments in both quantum computing (QC) and machine learning (ML). One of the ML paradigms that can be utilized to address challenging sequential decision-making issues is reinforcement learning (RL). It has been demonstrated that classical RL can successfully complete many difficult tasks. A leading method of building quantum RL agents relies on the variational quantum circuits (VQC). However, training QRL algorithms with VQCs requires significant amount of computational resources. This issue hurdles the exploration of various QRL applications. In this paper, we approach this challenge through asynchronous training of QRL agents. Specifically, we choose the asynchronous training of advantage actor-critic variational quantum policies. We demonstrate the results via numerical simulations that within the tasks considered, the asynchronous training of QRL agents can reach performance comparable to or superior to classical agents with similar model sizes and architectures. [ABSTRACT FROM AUTHOR]

This paper proposes a new, novel ranking method called Inverse-Reinforcement Learning Ranking. The main goal is to find a reward function representing the user's perceived utility after clicking on each result. It is necessary to utilize log information of all users' queries in the search engine dataset to reach this goal while assuming that the decisions (clicks) of the users are the best (optimal policy). The respective reward function is constructed using features extracted through a feature selection, and their corresponding weights are obtained by two optimization models, which are applicable for ranking results represented to the new users. In addition, new performance criteria were developed to illustrate the performance of the presented ranking method. To evaluate and test the proposed ranking algorithm, a real medium-sized dataset from a search engine is preprocessed and used in this research. Findings show promising results and decisive superiority over the default ranking method. It is illustrated that clicks on the top five results, top ten, and even the first results are remarkably improved by about 13–19% in all experiments, and the perplexity remarkably decreases by almost 23% after applying the ranking method. [ABSTRACT FROM AUTHOR]

In this paper, the measured input–output data sequences with pinning gain are proposed via the topology of multi-agent systems (MAS), where the requirement of reinforcement learning algorithm on internal state is avoided by pinning gain and the measured data of leader and neighbors. Besides, a data-based tracking state is given, which can be applied to MAS with different control matrices. According to the sequences and tracking state, a distributed control policy and corresponding reinforcement learning algorithm are proposed for the output synchronization. The proposed algorithm overcomes the shortcoming that previous algorithms can not be applied to MAS with different control matrices in the absence of model information and full-state vector. Finally, the effectiveness of proposed algorithm is verified by simulation examples. [ABSTRACT FROM AUTHOR]

• The AES power system intelligent energy scheduling environment is built. • The BI-LSTM-Att action network is propose by combining BI-LSTM with attention mechanism. • The DQN-CE algorithm is proposed by improving the DQN algorithm. • The experiments on an all-electric ferry is conducted using a variety of methods. At present, Ship energy scheduling based on deep reinforcement learning (DRL) is an important research direction in which Deep Q learning algorithm (DQN) has been successfully applied in the field of energy scheduling. In terms of the shortcomings of DQN in all electric ships(AES) energy scheduling, such that, the insufficient performance of using multiplayer perceptron(MLP) as action network and the degradation of convergence performance due to over estimation of Q value, in this paper, we propose a DQN cross entropy(DQN-CE) energy scheduling algorithm combing bi-directional LSTM and attention mechanism(BI-LSTM-Att). Instead of using MLP, firstly, we use BI-LSTM-Att as action network of DQN, and then an improved DRL algorithm called DQN-CE is proposed, that is, we add cross-entropy loss of predicted action of the next time for the target network and the action network to DQN. Finally, we conduct simulation experiments on an all-electric ferry, and the experimental results show that the energy scheduling algorithm proposed in this paper reduces the economic consumption by 4.11%, increases the utilization rate of energy storage system by 24.4%, and reduces the exploration time of agent training by 31.3% compared with the original DQN energy scheduling algorithm. Finally, the effectiveness and superiority of the algorithm were further verified in the new case study. [ABSTRACT FROM AUTHOR]

This paper proposes an online reinforcement learning(RL) for solving Stackelberg games with completely unknown dynamic systems. To deal with the hierarchical optimal control problem, the key is to find the solution of a two-level optimal control with the leader's optimal problem being constrained. Firstly, the leader–follower coupled Hamiltonian-Jacobi(HJ) equations subject to the follower's costate equation is derived. Secondly, an off-policy scheme is designed following the policy iteration(PI) algorithm based on model. The improved off-policy algorithm is given for obtaining the solution to the leader–follower coupled HJ equations. The algorithm is built without requiring any knowledge of the dynamic system, and the leader–follower optimal control policies are directly constructed by the extracted data. Meanwhile, the existence of the Stackelberg equilibrium is demonstrated. Lastly, NNs for each player are built, and NN learning is accomplished online with Kronecker product technique and vectorization, which simples the NN form and could decrease computation burden. Simulation examples are presented to demonstrate the proposed learning algorithm. [ABSTRACT FROM AUTHOR]

The lighting requirements are subjective and one light setting cannot work for all. However, there is little work on developing smart lighting algorithms that can adapt to user preferences. To address this gap, this paper uses fuzzy logic and reinforcement learning to develop an adaptive lighting algorithm. In particular, we develop a baseline fuzzy inference system (FIS) using the domain knowledge, generating light recommendation based on a set of intuitive rules. These rules, derived from existing literature, are based on environmental conditions i.e. daily glare index, and user information including age, activity, and chronotype. Through a feedback mechanism, the user interacts with the algorithm, correcting the algorithm output to their preferences. We interpret these corrections as rewards to a Q-learning algorithm, which tunes the FIS parameters online to match the user preferences. The Q-learning is a model-free learning algorithm that learns to act optimally by interacting with the user and the rewards it receives. This allows the proposed algorithm to work in a model-free manner, effectively handling the uncertainties that might arise from the individualistic preferences of users. To the authors' best knowledge, this algorithm is pioneering work in designing intelligent algorithms for personalized lighting control, featuring several elements of novelty, including the number of environmental and user-related inputs, the continuous control of light intensity as opposed to common on/off control, and the ability to learn user preferences. The algorithm is implemented in a real aircraft cabin and is evaluated in an extensive user study. The implementation results demonstrate that the developed algorithm possesses the capability to learn user preferences while successfully adapting to a wide range of environmental conditions and user characteristics. This underscores its viability as a potent solution for intelligent light management, featuring advanced learning capabilities. • Intelligent lighting algorithm with the ability to learn from and adapt to user preferences. • A fuzzy inference system tunes lighting based on environment and user traits. • The developed algorithm is tested and verified via an in-depth user study. [ABSTRACT FROM AUTHOR]

Grid-interactive building control poses a critical challenge in the context of grid modernization and decarbonization. Recently, various artificial intelligence-based optimal control approaches have been proposed, providing innovative solutions to optimal building control problems. However, researchers and practitioners are now confronted with a dilemma when selecting appropriate control strategies, ranging from model-based (knowledge-incorporating) to learning-based (data-driven) approaches, and hybrid methods combining them. Although each algorithm in existing literature claims superiority over specific baselines, their performance has never been systematically compared and analyzed, owing to the absence of a unified platform for comprehensive evaluation. To fill this knowledge gap, we identify and implement all state-of-the-art approaches within a modular training and evaluation framework, assessing their efficacy in a grid-interactive building control problem. In this paper, we also introduce a streamlined hybrid method that complements existing hybrid approaches. Our comparative study reveals and quantifies the advantages of hybrid methods: on average, they achieve near-optimal control while requiring less than 14% of the online computation of traditional model-based methods. To achieve this performance, they need as few as 2% of the training samples of purely learning-based methods. Finally, we provide insights into the merits, limitations, and implementation of each method to help researchers better understand the state of the art and future directions. The software framework implemented in this study is open-sourced to facilitate future research. [ABSTRACT FROM AUTHOR]

• The paper proposes a new algorithm for reinforcement learning for solving optimal control problems. • An application of biodiesel reactor optimization problem is used to illustrate the application of RL to optimal control problems. • The advantages and disadvantages of RL for solving optimal control problems are illustrated with a real-world application. Optimal control problems are one of the most challenging problems in optimization. This paper presents a new and efficient Reinforcement Learning approach to optimal control problems based on the Batch Q-learning algorithm. To improve the convergence of the RL algorithm, we use k-dimensional uniformity of advanced sampling procedures, namely employing Hamersley sequences (HSS). HSS is used to randomly sample the state variables and discrete controls from the action space for the RL optimal control problem. The Neural-fitted Q-iterative algorithm is applied to solve an optimal control problem for a first-order state dynamical system. A real-world application of optimal temperature profile determination for biodiesel production in a batch reactor is presented. We present the comparison of our HSS-RL algorithm with that of the maximum principle. [ABSTRACT FROM AUTHOR]

In this paper, a complementary learning scheme for experience transference of unknown continuous-time linear systems is proposed. The algorithm is inspired in the complementary learning properties that exhibit the hippocampus and neocortex learning systems via the striatum. The hippocampus is modelled as pattern-separated data of a human optimized controller. The neocortex is modelled as a Q-reinforcement learning algorithm which improves the hippocampus control policy. The complementary learning (striatum) is designed as an inverse reinforcement learning algorithm which relates the hippocampus and neocortex learning models to seek and transfer the weights of the hidden expert's utility function. Convergence of the proposed approach is analysed using Lyapunov recursions. Simulations are given to verify the proposed approach. [ABSTRACT FROM AUTHOR]

Deep Q-Networks algorithm (DQN) was the first reinforcement learning algorithm using deep neural network to successfully surpass human level performance in a number of Atari learning environments. However, divergent and unstable behaviour have been long standing issues in DQNs. The unstable behaviour is often characterised by overestimation in the Q -values, commonly referred to as the overestimation bias. To address the overestimation bias and the divergent behaviour, a number of heuristic extensions have been proposed. Notably, multi-step updates have been shown to drastically reduce unstable behaviour while improving agent's training performance. However, agents are often highly sensitive to the selection of the multi-step update horizon (n), and our empirical experiments show that a poorly chosen static value for n can in many cases lead to worse performance than single-step DQN. Inspired by the success of n -step DQN and the effects that multi-step updates have on overestimation bias, this paper proposes a new algorithm that we call 'Elastic Step DQN' (ES-DQN) to alleviate overestimation bias in DQNs. ES-DQN dynamically varies the step size horizon in multi-step updates based on the similarity between states visited. Our empirical evaluation shows that ES-DQN out-performs n -step with fixed n updates, Double DQN and Average DQN in several OpenAI Gym environments while at the same time alleviating the overestimation bias. [ABSTRACT FROM AUTHOR]

In this paper, we propose novel data-driven optimal dynamic controller design frameworks, via both state-feedback and output-feedback, for solving optimal output regulation problems of linear discrete-time systems subject to unknown dynamics and unmeasurable disturbances using reinforcement learning (RL). Fundamentally different from existing research on optimal output regulation problems and RL, the proposed procedures can determine both the optimal control gain and the optimal dynamic compensator simultaneously instead of presetting a non-optimal dynamic compensator. Moreover, we present incremental dataset-based RL algorithms to learn the optimal dynamic controllers that do not require the measurements of the external disturbance and the exostate during learning, which is of great practical importance. Besides, we show that the proposed incremental dataset-based learning methods are more robust to a class of measurement noises with arbitrary magnitudes than routine RL algorithms. Comprehensive simulation results validate the efficacy of our methodologies. [ABSTRACT FROM AUTHOR]

• Peoples who're specially affected with heart sickness and it's far one of the man-kill illnesses in the international level. • Most of researchers to awareness at the prediction, clustering, rule generation, decision tree and machine learning algorithm for figuring out and predicting the danger of the sufferers primarily based on the medical information. • The researcher can pick out the time and reminiscence wanted for the execution. • There are many different types of machine learning algorithms are categorized into three important classifications namely unsupervised learning, supervised learning and reinforcement learning. • This research paper includes the six elements to evaluate the overall performance of K-Means, Navie-Bayes and enhanced PSNB-IMR Algorithm with various parameters. Peoples who're specially affected with heart sickness and it's far one of the man-kill illnesses in the international level. Most of researchers to awareness at the prediction, clustering, rule generation, decision tree and machine learning algorithm for figuring out and predicting the danger of the sufferers primarily based on the medical information. The overall performances of the crucial functions are based on the machine-learning concept. By studying the algorithm, the researcher can pick out the time and reminiscence wanted for the execution. As such, there are many different types of machine learning algorithms are categorized into three important classifications namely unsupervised learning, supervised learning and reinforcement learning. Unsupervised learning consists of all varieties of clustering algorithms at the same time as supervised learning algorithm consists of all of the category strategies. But the author is considered the two algorithms are Supervised and Unsupervised learning algorithm to examine the overall performance. This research paper includes the six elements to evaluate the overall performance of K-Means, Navie-Bayes and enhanced PSNB-IMR Algorithm with various parameters. [ABSTRACT FROM AUTHOR]

The free-flying space manipulators present challenges in controlling the motions of both the spacecraft bus and the manipulator, because of the highly-coupling system dynamics and the unknown space environment disturbances. Although the sliding mode controllers are robust to the unknown disturbances and system uncertainties, the chattering effect could affect the pointing accuracy and the lifetime of the actuators. This paper first introduces the dynamics of a CuBot, which is a 3-rigid-link manipulator based on the CubeSat platform. To maintain the robustness while decreasing the chattering effect, an innovative reinforcement learning based fuzzy adaptive sliding mode controller is proposed. To maintain the robustness while reducing the chattering effect, an innovative reinforcement learning based fuzzy adaptive sliding mode controller is proposed. The switching gain is updated to estimate the lumped upper bound of the system uncertainties and the unknown disturbances, and then a new fuzzy logic adaptive law is applied on the switching gain to decrease the chattering effects. On top of that, the fuzzy logic rules are tuned by an innovative modified reinforcement learning mechanism to achieve the better tracking performance. The uniformly ultimately bounded tracking errors are guaranteed by the proposed control scheme, and the effectiveness is validated by the simulation results. • A space manipulator is designed on the 8U CubeSat bus. • A modified reinforcement learning algorithm is designed to tune the fuzzy logic rules. • A fuzzy adaptive sliding mode controller is designed to reduce the chattering effect. • The new algorithm is compared to a conventional sliding mode controller. • The performance of the new algorithm is verified by simulation. [ABSTRACT FROM AUTHOR]

We explored the problem about function approximation error and complex mission adaptability in multi-agent deep reinforcement learning. This paper proposes a new multi-agent deep reinforcement learning algorithm framework named multi-agent time delayed deep deterministic policy gradient. Our work reduces the overestimation error of neural network approximation and variance of estimation result using dual-centered critic, group target network smoothing and delayed policy updating. According to experiment results, it improves the ability to adapt complex missions eventually. Then, we discuss that there is an inevitable overestimation issue about existing multi-agent algorithms about approximating real action-value equations with neural network. We also explain the approximate error of equations in the multi-agent deep deterministic policy gradient algorithm mathematically and experimentally. Finally, the application of our algorithm in the mixed cooperative competition experimental environment further demonstrates the effectiveness and generalization of our algorithm, especially improving the group's ability of adapting complex missions and completing more difficult missions. [ABSTRACT FROM AUTHOR]

Hyperparameter tuning is critical for the performance of machine learning algorithms. However, a noticeable limitation is the high computational cost of algorithm evaluation for complex models or for large datasets, which makes the tuning process highly inefficient. In this paper, we propose a novel model-based method for efficient hyperparameter optimization. Firstly, we frame this optimization process as a reinforcement learning problem and then employ an agent to tune hyperparameters sequentially. In addition, a model that learns how to evaluate an algorithm is used to speed up the training. However, model inaccuracy is further exacerbated by long-term use, resulting in collapse performance. We propose a novel method for controlling the model use by measuring the impact of the model on the policy and limiting it to a proper range. Thus, the horizon of the model use can be dynamically adjusted. We apply the proposed method to tune the hyperparameters of the extreme gradient boosting and convolutional neural networks on 101 tasks. The experimental results verify that the proposed method achieves the highest accuracy on 86.1% of the tasks, compared with other state-of-the-art methods and the average ranking of runtime is significant lower than all methods by using the predictive model. [ABSTRACT FROM AUTHOR]

Because of the sparse reward and the sequence of the complex multi-step task, there is a big challenge in reinforcement learning, that is, the agent needs to implement several consecutive sequential steps to complete the whole task without the intermediate reward. Reward Shaping and Curriculum Learning algorithms are always used to solve this challenge, but Reward Shaping is prone to sub-optimal policies and Curriculum Learning easily suffers from the catastrophic forgetting problem. In this paper, we propose a novel algorithm called Parallel Multi-Environment Shaping (PMES), where several sub-environments are built based on human knowledge to make the agent aware of the importance of intermediate steps, each of which corresponds to a key intermediate step. Specifically, the learning agent is trained under these parallel multiple environments including the original environment and several sub-environments by synchronous advantage actor-critic algorithm. And the PMES algorithm has the mechanism of adaptive reward shaping to adjust the reward function. In this way, PMES algorithm effectively incorporates human experience by multiple different environments rather than only shaping the reward function, which combines the benefits of Reward Shaping and Curriculum Learning algorithms while avoiding their drawbacks. Extensive experiments on the mini-game 'Build Marines' of StarCraft II environment show that our proposed algorithm is more effective than Reward Shaping, Curriculum Learning, and PLAID algorithms, which is almost close to the level of human Grandmaster. And compared with the existing work, it takes less time and computing resources to reach a good result. [ABSTRACT FROM AUTHOR]

This paper presents an adaptive neuro-fuzzy PID controller based on twin delayed deep deterministic policy gradient (TD3) algorithm for nonlinear systems. In this approach, the observation of the environment is embedded with information of a multiple input single output (MISO) fuzzy inference system (FIS) and have a specially defined fuzzy PID controller in neural network (NN) formation acting as the actor in the TD3 algorithm, which achieves automatic tuning of gains of fuzzy PID controller. From the control perspective, the controller combines the merits of both FIS and PID controller and utilizes reinforcement learning algorithm for optimizing parameters. From the reinforcement learning point of view, embedding the prior knowledge into the fuzzy PID controller incorporated in the actor network helps reduce the learning difficulty in the training process. The proposed method was tested on the cart-pole system in simulation environment with comparison of a linear PID controller, which demonstrates the robustness and generalization of the proposed approach. [ABSTRACT FROM AUTHOR]

Aeroengine fault detection is an important means to ensure flight safety. The application premise of data driven fault detection method is that all data come from the same distribution. However, this assumption is invalid in the actual engine fault detection, because the engine state will change with the increase of operating time, and the collected data will also have distribution differences. Due to the high cost of collecting engine data, it is difficult to collect enough data from the current state. Fortunately, transfer learning can transfer data information from other fields to the target field, thus alleviating the problem of data scarcity in the target domain. Therefore, from the idea of transfer learning, this paper proposes a cross domain aeroengine fault detection method, viz. transfer learning based on kernel perception algorithm with uneven margins (TL-KPAUM). The proposed method is divided into two stages. In the first stage, KPAUM is trained using the data in the source domain to extract the information in the source domain. In the second stage, the data in the target domain is used to realize the adaptation to the target domain. Compared with several baseline methods, TL-KPAUM has better performance when the amount of data in the target domain is small. Finally, using the simulation data of turbofan engine, the fault detection experiments of degraded state are designed, and the results show that this method is effective. [Display omitted] • A transfer learning algorithm is proposed for aeroengine fault detection. • The proposed method is applied to engine fault detection in different domains. • Simulation results show the effectiveness of this method compared with other methods. • The proposed method has good performance in the case of sparse data in the target domain. [ABSTRACT FROM AUTHOR]

Fuzzy inference systems with reinforcement learning are currently being used in differential games to train agents with no prior experience. However, the reinforcement learning algorithms based on actor-critic structure have a drawback that the policy is depended on a probability distribution. In this paper, a novel fuzzy deterministic policy gradient algorithm is introduced and applied to classical 1-vs-1 constant-velocity pursuit-evasion differential games. The key goal is to self-learn the optimal strategy in the continuous action domain and obtain a specific physical meaning of the fuzzy rules. The novel proposed algorithm is based on the deterministic policy gradient theorem and the agent learns the near-optimal strategy under the actor-critic structure. The fuzzy inference system is applied as approximators so that the specific physical meaning can be obtained by the linguistic fuzzy rules. Furthermore, the proposed algorithm is applied to solve the decision-making problem of pursuit-evasion differential games. The result is compared with other existing algorithms and it elucidates that the proposed algorithm outperforms the precision and convergence efficiency. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning systems are now a hot topic in Machine Learning for their effectiveness in many complex tasks, but their application in safety-critical domains (e.g., robot control or self-autonomous driving) remains dangerous without mechanism to detect and prevent risk situations. In Deep RL, such risk is mostly in the form of adversarial attacks, which introduce small perturbations to sensor inputs with the aim of changing the network-based decisions and thus cause catastrophic situations. In the light of these dangers, a promising line of research is that of providing these Deep RL algorithms with suitable defenses, especially when deploying in real environments. This paper suggests that this line of research could be greatly improved by the concepts from the existing research field of Safe Reinforcement Learning, which has been postulated as a family of RL algorithms capable of providing defenses against many forms of risks. However, the connections between Safe RL and the design of defenses against adversarial attacks in Deep RL remain largely unexplored. This paper seeks to explore precisely some of these connections. In particular, this paper proposes to reuse some of the concepts from existing Safe RL algorithms to create a novel and effective instance-based defense for the deployment stage of Deep RL policies. The proposed algorithm uses a risk function based on how far a state is from the state space known by the agent, that allows identifying and preventing adversarial situations. The success of the proposed defense has been evaluated in 4 Atari games. [ABSTRACT FROM AUTHOR]

Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as MS - ADTD) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of MS - ADTD , the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments. • This paper proposes a distributed adaptive TD learning algorithm under Markovian sampling. • It theoretically analyzes the non-asymptotic convergence performance of the proposed algorithm. • It verifies the performance of the proposed algorithm by experiments. [ABSTRACT FROM AUTHOR]

Multiobjective evolutionary algorithms (MOEAs) have gained much attention due to their high effectiveness and efficiency in solving multiobjective optimization problems (MOPs). However, when solving MOPs, it is important but difficult to maintain a good balance of exploration and exploitation. In addition, some reference point based MOEAs with fixed reference points perform poorly on MOPs with irregular frontiers. Therefore, this paper proposes a new multistrategy multiobjective differential evolutionary (DE) algorithm, named RLMMDE. In RLMMDE, a multistrategy and multicrossover DE optimizer is utilized to alleviate the exploration and exploitation dilemma. An adaptive reference point activation mechanism based on RL is proposed to activate the adaptive adjustment of reference points. Moreover, a reference point adaptation method is proposed to improve the performance of RLMMDE on irregular frontier problems. Experimental results of RLMMDE tested on some benchmark test suites (i.e., ZDT, DTLZ, UF, WFG, and LSMOP) and two practical mixed-variable optimization problems show that the algorithm outperforms some advanced MOEAs. [ABSTRACT FROM AUTHOR]

Buying and selling stocks are the main activities of investment in the financial markets. During several years, financial experts have designed various indices and indicators to develop intelligent methods to decide whether buy or sell or no trade a specific stock. In this paper, the league championship algorithm (LCA) is adapted and equipped with a network structure (which provides the implicit memory function, compact structure and gives the ability of reusing nodes) for stock trading rule extraction process. The proposed algorithm is able to extract and save various stock trading rules for various kinds of stock market conditions. Each individual in LCA makes role as a sport team which could learn from weaknesses and strengths of others to enhance its performance. The network structure used for each individual is equipped with reinforcement learning to strength the searching power and extracting better rules. The backward Q-learning (BQL), which is a combination of Sarsa learning and Q-learning algorithms, is also hybridized with LCA to increase the learning speed to extract better rules. A trend clustering technique is also developed to determine uptrends and downtrends and to remedy delay in the trend direction reflection along with moving average of stock prices. Finally, to use the trading rules extracted and saved in the rule pools, a new approach through average matching degree is introduced which could create trinary signals of buying, selling, and no trade. The effectiveness of the proposed approach is tested on a set of 20 companies, which have high liquidity and large market capitalization, chosen from various fields of Tehran Stock Exchange. The used historical price data was adjusted through factors of dividends and capital increase. Computational tests and simulations indicate that our hybrid method clearly offers higher profits and lower losses than those of genetic network programming method with rule accumulation (GNP-RA) on all stocks and those of the Buy&Hold strategy on 17 out of 20 instances (85%), especially when the price uptrends are sharp. [ABSTRACT FROM AUTHOR]

The Successor Feature framework for Reinforcement Learning algorithms improves task transfer by decomposing the learned state–action value function. The decomposition involves two components, one that captures future-expected state features and the other that models the task-related reward structure. However, successful transfer between tasks depends heavily on how the reward function changes, possibly leading to failure of the original Successor Feature formulation. This paper proposes the Dynamic Successor Feature framework, DynSF, by extending the mathematical formulation of the original Successor Feature framework to center around a learned state-transition model. Under this formulation, the state-transition model dynamically induces the acting policy. The flexibility of DynSF also extends to the architecture, requiring only a state-transition model and a small vector of parameters. This architecture provides immense flexibility in the choice of the model used to learn the state-transition model. The DynSF framework is evaluated and compared to other baseline algorithms through several experiments in a continuous grid world environment, a robotic Reacher, and pixels in the Doom environment. • DynSF enables a state-transition model to be used in Reinforcement Learning. • Policy and discount factors can be parameterized dynamically. • Different supervised ML algorithms can be used to learn the state transition model. • DynSF can be used for guided exploration and transfer learning. • DynSF performs better during task transfer than the Successor Feature framework. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) is a learning paradigm that can be useful in a wide variety of real-world applications. However, its applicability to complex problems remains problematic due to different causes. Particularly important among these are the high quantity of data required by the agent to learn useful policies and the poor scalability to high-dimensional problems due to the use of local approximators. This paper presents a novel RL algorithm, called online fitted policy iteration (OFPI), that steps forward in both directions. OFPI is based on a semi-batch scheme that increases the convergence speed by reusing data and enables the use of global approximators by reformulating the value function approximation as a standard supervised problem. The proposed method has been empirically evaluated in three benchmark problems. During the experiments, OFPI has employed a neural network trained with the extreme learning machine algorithm to approximate the value functions. Results have demonstrated the stability of OFPI using a global function approximator and also performance improvements over two baseline algorithms (SARSA and Q-learning) combined with eligibility traces and a radial basis function network. [ABSTRACT FROM AUTHOR]

In this paper we proposed reinforcement learning algorithms with the generalized reward function. In our proposed method we use Q-learning1 and SARSA1 algorithms with generalised reward function to train the reinforcement learning agent. We evaluated the performance of our proposed algorithms on Real Time Strategy (RTS) game called BattleCity. There are two main advantages of having such an approach as compared to other works in RTS. (1) We can ignore the concept of a simulator which is often game specific and is usually hard coded in any type of RTS games (2) our system can learn from interaction with any opponents and quickly change the strategy according to the opponents and do not need any human traces as used in previous works. [ABSTRACT FROM AUTHOR]

Simultaneous multithreading (SMT) improves the performance of superscalar CPUs by exploiting thread-level parallelism with shared entries for better utilization of resources. A key issue for this out-of-order execution is that the occupancy latency of a physical rename register can be undesirably long due to many program execution-dependent factors that result in performance degradation. Such an issue becomes even more problematic in an SMT environment in which these registers are shared among concurrently running threads. Smartly managing this critical shared resource to ensure that slower threads do not block faster threads' execution is essential to the advancement of SMT performance. In this paper, an actor–critic style reinforcement learning (RL) algorithm is proposed to dynamically assigning an upper-bound (cap) of the rename registers any thread is allowed to use according to the threads' real-time demand. In particular, a critic network projects the current Issue Queues (IQ) usage, register file usage, and the cap value to a reward; an actor network is trained to project the current IQ usage and register file usage to the optimal real-time cap value via ascending the instructions per cycle (IPC) gradient within the trajectory distribution. The proposed method differs from the state-of-the-art (Wang and Lin, 2018) as the cap for the rename registers for each thread is adjusted in real-time according to the policy and state transition from self-play. The proposed method shows an improvement in IPC up to 162.8% in a 4-threaded system, 154.8% in a 6-threaded system and up to 101.7% in an 8-threaded system. The code is now available open source at https://github.com/98k-bot/RL-based-SMT-Register-Renaming-Policy. • The first reinforcement learning approach for effective resource utilization in SMT. • Continuous action space for fine-grained and accurate action space exploration. • Encouraging the convergence of the model via forming a lower bound of the policy. • A self-play mechanism compensates for the long time-scale variance of the data. [ABSTRACT FROM AUTHOR]

This paper proposes AutoGCOP, a new general framework for automated design of local search algorithms. In a recently established General Combinatorial Optimisation Problem (GCOP) model, the problem of algorithm design itself is defined as a combinatorial optimisation problem. AutoGCOP defines a general framework to optimise the composition of elementary algorithmic components as decision variables in GCOP. By modelling various well-known local search meta-heuristics within a general framework, AutoGCOP supports automatic design of new novel algorithms which may be highly different from those manually designed in the literature. Within the consistent AutoGCOP framework, various elementary algorithmic components are analysed for solving the benchmark vehicle routing problem with time window constraints and different characteristics. Furthermore, two learning models based on reinforcement learning and Markov chain are investigated to learn and enhance the compositions of algorithmic components towards the automated design of search algorithms. The Markov chain model presents superior performance learning the compositions of algorithmic components during the search, demonstrating its effectiveness designing new algorithms automatically. • A framework for automated design of local search algorithms. • Various local search algorithms are modelled within the framework. • The vehicle routing problems are used as the domain problem. • Performance of elementary algorithmic components is analysed. • Two learning models based on reinforcement learning and Markov chain are compared. [ABSTRACT FROM AUTHOR]

• We develop a multi-objective cooperative deep reinforcement learning based CAV car following control strategy for a mixed traffic. • We embed the concept of equilibrium point inside deep reinforcement learning which facilitates algorithm convergence and stability of mixed traffic. • We embed the HDV characteristic inside the environment of deep reinforcement learning to further improve the performance of control. • We analyze the vehicle sequencing impact on the mixed traffic. This paper proposes a cooperative strategy of connected and automated vehicles (CAVs) longitudinal control for a mixed connected and automated traffic environment based on deep reinforcement learning (DRL) algorithm, which enhances the string stability of mixed traffic, car following efficiency, and energy efficiency. Since the sequences of mixed traffic are combinatory, to reduce the training dimension and alleviate communication burdens, we decomposed mixed traffic into multiple subsystems where each subsystem is comprised of human-driven vehicles (HDV) followed by cooperative CAVs. Based on that, a cooperative CAV control strategy is developed based on a deep reinforcement learning algorithm, enabling CAVs to learn the leading HDV's characteristics and make longitudinal control decisions cooperatively to improve the performance of each subsystem locally and consequently enhance performance for the whole mixed traffic flow. For training, a distributed proximal policy optimization is applied to ensure the training convergence of the proposed DRL. To verify the effectiveness of the proposed method, simulated experiments are conducted, which shows the performance of our proposed model has a great generalization capability of dampening oscillations, fulfilling the car following and energy-saving tasks efficiently under different penetration rates and various leading HDVs behaviors. [ABSTRACT FROM AUTHOR]

• The combination of mobile devices and edge computing reduces the total computing overhead and makes resource allocation reasonable. • Evaluate system performance by the weighted sum of delay and energy consumption. • Change the parameter update method to accelerate the convergence speed of the deep reinforcement learning algorithm. • A neural network is used to approximate the behavior value function, and a target q network is used to update the target. With the arrival of the 5th generation mobile networks (5 G) era, the data needed by mobile devices (MDs) is explosively growing. High-consumption, low-latency applications are huge challenges for resource-constrained Internet of things (IoT) devices. Mobile edge computing overcomes the limitations of computing resources on MDs by offloading tasks generated by MDs and assigning them to nearby MEC servers. Therefore, mobile edge computing (MEC) becomes important. This paper presents a task offloading strategy for the multi-device multi-server system. To meet the task requirements of different MDs, we formulate an overhead minimization problem to optimize the delay and energy consumption of the system. We propose the Double Deep Q Network (Double-DQN) algorithm to perform location selection strategies for tasks generated on the mobile devices and allocate respective computing resources. Simulation results show that the algorithm can allocate resources reasonably and reduce the overhead of the entire system. [Display omitted] [ABSTRACT FROM AUTHOR]

• Additional reward parameter is added to the learning algorithm. • The replicator equation is extended from single-state to transition-state. • The algorithm and its dynamic are demonstrated on a 2 states battle of sexes game. Agent-based evolutionary game theory studies the dynamics of the autonomous agents. It is important for application that relies on the agents to perform the automated tasks. Since the agents make their own decision, therefore the stability of the interaction needs to be comprehended. The current state of the art in agent-based replicator dynamics are piecewise and state-coupled replicator dynamics which focus on joint-action single-state reward. This paper introduces additional reward parameter to the learning algorithm, extends the replicator dynamics to joint-action transition-state reward and shows that it can be changed to single-state reward and independent-action reward. The replicator equation is expressed based on the tree diagram approach and is verified with the numerical simulation in a two states battle of sexes coordination game for various types of rewards. The numerical results are consistent with the phase portraits generated by the replicator equation and are able to provide some general insights to the coordination game such as the number of convergence points, the rate of convergence and the effect of initial points on the convergence. [ABSTRACT FROM AUTHOR]

With the ever-increasing operating mileage of urban rail transit systems, a large amount of energy is consumed for train operation, and the energy-saving for train operation has become an attractive topic in the research community in recent years. In order to minimize the net energy consumption (i.e., the difference between the traction energy and the reused regenerative energy), an energy-efficient train operation method using the cooperative control approach is proposed in this paper. Firstly, a set of mathematical models for the cooperative control are formulated with considering the transmission mechanism of regenerative energy. Then, a multi-agent reinforcement learning algorithm is designed to obtain the cooperative driving strategy for trains. In this algorithm, the global Q-function is decomposed into several local Q-functions by using value function factorization method. A proposed update method is applied to update the local Q-functions according to the transmission mechanism of regenerative energy. Finally, the case studies built upon a metro line is performed to illustrate the effectiveness of the proposed method. According to the simulation results, the net energy consumption is reduced by more than 11.1% compared to the method in which the driving strategy of each train is independently optimized. • An innovative framework of the energy-efficient operation by cooperative control among trains. • An improved multi-agent reinforcement learning algorithm based on the weighted decomposition of individual reward. • The energy-efficient ratio can reach more than 11.1%. [ABSTRACT FROM AUTHOR]

Off-grid microgrids are receiving a growing interest for rural electrification purposes in developing countries due to their ability to ensure affordable, sustainable and reliable energy services. Off-grid microgrids rely on renewable energy sources (RES) coupled with storage systems to supply the electrical consumption. The inherent uncertainty introduced by RES as well as the stochastic nature of the electrical demand in rural contexts pose significant challenges to the efficient control of off-grid microgrids throughout their entire life span. In this paper, we address the lifelong control problem of an isolated microgrid. We categorize the set of changes that may occur over its life span in progressive and abrupt changes. We propose a novel model-based reinforcement learning algorithm that is able to address both types of changes. In particular, the proposed algorithm demonstrates generalisation properties, transfer capabilities and better robustness in case of fast-changing system dynamics. The proposed algorithm is compared against a rule-based policy and a model predictive controller with look-ahead. The results show that the trained agent is able to outperform both benchmarks in the lifelong setting where the system dynamics are changing over time. • Open-source reinforcement learning framework for the lifelong off-grid microgrid control. • Changes that may occur over the lifetime of the microgrid are considered. • A novel model-based reinforcement learning algorithm is proposed. • The proposed algorithm performs well in changing conditions. [ABSTRACT FROM AUTHOR]

Controlling heating, ventilation and air-conditioning (HVAC) systems is crucial to improving demand-side energy efficiency. At the same time, the thermodynamics of buildings and uncertainties regarding human activities make effective management challenging. While the concept of model-free reinforcement learning demonstrates various advantages over existing strategies, the literature relies heavily on value-based methods that can hardly handle complex HVAC systems. This paper conducts experiments to evaluate four actor-critic algorithms in a simulated data centre. The performance evaluation is based on their ability to maintain thermal stability while increasing energy efficiency and on their adaptability to weather dynamics. Because of the enormous significance of practical use, special attention is paid to data efficiency. Compared to the model-based controller implemented into EnergyPlus, all applied algorithms can reduce energy consumption by at least 10% by simultaneously keeping the hourly average temperature in the desired range. Robustness tests in terms of different reward functions and weather conditions verify these results. With increasing training, we also see a smaller trade-off between thermal stability and energy reduction. Thus, the Soft Actor Critic algorithm achieves a stable performance with ten times less data than on-policy methods. In this regard, we recommend using this algorithm in future experiments, due to both its interesting theoretical properties and its practical results. • Comparison of four model-free actor-critic reinforcement learning algorithms in a data-centre simulation. • Each of the applied algorithms can learn the task and reduce energy consumption. • The conflict between thermal stability and energy savings is reduced during the training. • The algorithms are robust to changing reward functions and weather conditions. • Soft Actor Critic shows the highest data efficiency and is recommended for applications. [ABSTRACT FROM AUTHOR]

• An image process pipeline is introduced to locate guava fruits and obstacles. • A recurrent deep reinforcement learning algorithm is proposed to predict collision-free paths. • The robot is trained in a simulation environment and transferred to the real world directly. • The robot only needs 29 ms to plan a collision-free path with a success rate of 90.9%. In unstructured orchard environments, picking a target fruit without colliding with neighboring branches is a significant challenge for guava-harvesting robots. This paper introduces a fast and robust collision-free path-planning method based on deep reinforcement learning. A recurrent neural network is first adopted to remember and exploit the past states observed by the robot, then a deep deterministic policy gradient algorithm (DDPG) predicts a collision-free path from the states. A simulation environment is developed and its parameters are randomized during the training phase to enable recurrent DDPG to generalize to real-world scenarios. We also introduce an image processing method that uses a deep neural network to detect obstacles and uses many three-dimensional line segments to approximate the obstacles. Simulations show that recurrent DDPG only needs 29 ms to plan a collision-free path with a success rate of 90.90%. Field tests show that recurrent DDPG can increase grasp, detachment, and harvest success rates by 19.43%, 9.11%, and 10.97%, respectively, compared to cases where no collision-free path-planning algorithm is implemented. Recurrent DDPG strikes a strong balance between efficiency and robustness and may be suitable for other fruits. [ABSTRACT FROM AUTHOR]

Hyperparameter optimization is critical for the performance of machine learning algorithms. Significant efforts have been dedicated to improve the final accuracy of algorithm by hyperparameter tuning. However, some indicators (such as latency, cpu utilization) are also very important in the actual environment. In this paper, we propose a novel method EMORL (E ffective M ulti- O bjective R einforcement L earning) based on multi-objective reinforcement learning for hyperparameter optimization to solve the above limitations. Specifically, we extend hyperparameter optimization problem to the reinforcement learning framework and employ an agent to select hyperparameters sequentially, and design a scalarization function that combines accuracy and latency as a multi-objective reward to guide the policy update. To improve the efficiency of hyperparameter optimization, previously successful configuration is reused for reshaping the advantage function. In the experiment, we apply the proposed method to tune the hyperparameters of the eXtreme Gradient Boosting on 101 tasks and convolutional neural networks on 2 tasks. The experimental results demonstrate that the proposed method is better than other methods in most tasks, especially in terms of latency. In addition, we verify the various components of the proposed method through ablation experiments. [ABSTRACT FROM AUTHOR]

Traffic signal control helps to reduce traffic congestion and thus has been studied for a few decades. Algorithms for multi-intersection traffic signal control that adopt hierarchical deep reinforcement learning have been shown to achieve state-of-the-art performance. For these methods, a two-level hierarchical structure is generally used, in which the low-level policies employ latent goals induced by the high-level policies. Existing research either designs the latent goals manually or acquires the latent goals from the environment. However, these latent goals are not optimal, which leads to the non-optimal low-level policies. To improve this, we propose a learned-goal soft actor–critic (LSAC) algorithm, by which the optimal latent goals are automatically learned and then are used in low-level policies. We then propose a semi-decentralized feudal multi-agent (SFM) framework, which can alleviate the problem that existing multi-agent framework faces, i.e., the fast growing state space with the increase of the number of agents. Combining the above two proposed methods, an SFM-LSAC algorithm is proposed in this paper for multi-intersection traffic signal control. Other techniques are also incorporated into the SFM-LSAC algorithm such as attention mechanism. Experimental results in three real-world traffic scenarios illustrate that the SFM-LSAC algorithm outperforms other four state-of-the-art multi-intersection traffic signal control algorithms, i.e., reduces the average intersection delay, the average queue length, the average travel time, meanwhile improves the average flow and average travel speed, thus could be a new promising algorithm for multi-intersection traffic signal control. • A representation learning algorithm is proposed to learn the optimal latent goals. • A learned-goal soft actor–critic algorithm is proposed as one traffic-signal agent. • A semi-decentralized feudal multi-agent framework is proposed to lower state space. • Based on above methods, the whole algorithm is proposed for traffic signal control. • Experiments show the whole algorithm outperforms the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

• Genetic-State Grouping Algorithm guarantees enhancing the performance of RL agents. • The genetic algorithm has successfully been combined with Monte Carlo Tree Search. • Video game provides a valid proving ground for testing AI's capability. Although Reinforcement learning has already been considered one of the most important and well-known techniques of machine learning, its applicability remains limited in the real-world problems due to its long initial learning time and unstable learning. Especially, the problem of an overwhelming number of the branching factors under real-time constraint still stays unconquered, demanding a new method for the next generation of reinforcement learning. In this paper, we propose Genetic State-Grouping Algorithm based on deep reinforcement learning. The core idea is to divide the entire set of states into a few state groups. Each group consists of states that are mutually similar, thus representing their common features. The state groups are then processed with the Genetic Optimizer, which finds outstanding actions. These steps help the Deep Q Network avoid excessive exploration, thereby contributing to the significant reduction of initial learning time. The experiment on the real-time fighting video game (FightingICE) shows the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

In fact, optimizing path within short computation time still remains a major challenge for mobile robotics applications. In path planning and obstacles avoidance, Q-Learning (QL) algorithm has been widely used as a computational method of learning through environment interaction. However, less emphasis is placed on path optimization using QL because of its slow and weak convergence toward optimal solutions. Therefore, this paper proposes an Efficient Q-Learning (EQL) algorithm to overcome these limitations and ensure an optimal collision-free path in less possible time. In the QL algorithm, successful learning is closely dependent on the design of an effective reward function and an efficient selection strategy for an optimal action that ensures exploration and exploitation. In this regard, a new reward function is proposed to initialize the Q-table and provide the robot with prior knowledge of the environment, followed by a new efficient selection strategy proposal to accelerate the learning process through search space reduction while ensuring a rapid convergence toward an optimized solution. The main idea is to intensify research at each learning stage, around the straight-line segment linking the current position of the robot to T a r g e t (optimal path in terms of length). During the learning process, the proposed strategy favors promising actions that not only lead to an optimized path but also accelerate the convergence of the learning process. The proposed EQL algorithm is first validated using benchmarks from the literature, followed by a comparison with other existing QL -based algorithms. The achieved results showed that the proposed EQL gained good learning proficiency; besides, the training performance is significantly improved compared to the state-of-the-art. Concluded, EQL improves the quality of the paths in terms of length, computation time and robot safety, furthermore outperforms other optimization algorithms. • The mobile robot path optimization problem is handled and modeled. • An Efficient Q-Learning (EQL) algorithm is proposed. • In EQL , a new definition of states space and actions space is proposed. • New reward function is proposed to initialize Q-table. • Learning process is sped up by exploiting a new efficient selection strategy. • Results on benchmarks from literature demonstrate EQL efficiency and superiority. [ABSTRACT FROM AUTHOR]

In the field of unmanned surface vehicles, intelligent collision avoidance technology is essential to ensure the safety of navigating. In this paper, the problem of avoiding moving boats for USVs under the constraints of COLREGs is studied. A COLREGs intelligent collision avoidance (CICA) algorithm based on deep reinforcement learning is proposed, which can automatically extract state features by using powerful deep neural networks. The reward function is designed, which ensures that the USV navigates to the target while obeying COLREGs to avoid dynamic obstacles. A method is proposed to track the current network weight to update the target network weight, which improves the stability of the algorithm in learning the optimal strategy. It is shown that the CICA algorithm converges with fewer training times through ε-greedy with both decaying ε and reward threshold than other three strategies. By comparing the CICA algorithm with the artificial potential field method and the velocity obstacle method, it is concluded that the CICA algorithm is superior to the other two algorithms. • A intelligent collision avoidance algorithm for underactuated USVs is proposed based on deep reinforcement learning. • The USV has the ability to avoid dynamic obstacles under the control of this algorithm, and obeys COLREGs. • The algorithm can deal with complex scenarios without defining a complete collision avoidance control law. [ABSTRACT FROM AUTHOR]

• Apply a state-of-the-art deep reinforcement learning algorithm to the PM problem. • Formulate the PM problem as an MDP with guidance of domain knowledge. • Use a data-driven modeling method to build a fast simulator for learning. • The DRL agent learns to perform group maintenance and opportunistic maintenance. • Proposed method outperforms age-dependent policy and opportunistic policy. In the manufacturing industry, the preventive maintenance (PM) is a common practice to reduce random machine failures by replacing/repairing the aged machines or parts. The decision on when and where the preventive maintenance needs to be carried out is nontrivial due to the complex and stochastic nature of a serial production line with intermediate buffers. In order to improve the cost efficiency of the serial production lines, a deep reinforcement learning based approach is proposed to obtain PM policy. A novel modeling method for the serial production line is adopted during the learning process. A reward function is proposed based on the system production loss evaluation. The algorithm based on the Double Deep Q-Network is applied to learn the PM policy. Using the simulation study, the learning algorithm is proved effective in delivering PM policy that leads to an increased throughput and reduced cost. Interestingly, the learned policy is found to frequently conduct "group maintenance" and "opportunistic maintenance", although their concepts and rules are not provided during the learning process. This finding further demonstrates that the problem formulation, the proposed algorithm and the reward function setting in this paper are effective. [ABSTRACT FROM AUTHOR]

Fixed-Wing UAVs (Unmanned Aerial Vehicles) flocking is still a challenging problem due to the kinematics complexity and environmental dynamics. In this paper, we solve the leader–followers flocking problem using a novel deep reinforcement learning algorithm that can generate roll angle and velocity commands by training an end-to-end controller in continuous state and action spaces. Specifically, we choose CACLA (Continuous Actor–Critic Learning Automation) as the base algorithm and we use the multi-layer perceptron to represent both the actor and the critic. Besides, we further improve the learning efficiency by using the experience replay technique that stores the training data in the experience memory and samples from the memory as needed. We have compared the performance of the proposed CACER (Continuous Actor–Critic with Experience Replay) algorithm with benchmark algorithms such as DDPG and double DQN in numerical simulation, and we have demonstrated the performance of the learned optimal policy in semi-physical simulation without any parameter tuning. • We develop a DRL-based framework for the flocking of fixed-wing UAVs. • We propose an actor–critic RL algorithm for UAV flocking in continuous spaces. • The learned policy can be directly transferred to the semi-physical simulation. [ABSTRACT FROM AUTHOR]