Search

In this paper, we present a Q-learning framework for solving finite-horizon zero-sum game problems involving the H ∞ control of linear system without knowing the dynamics. Research in the past mainly focused on solving problems in infinite horizon with completely measurable state. However, in the practical engineering, the system state is not always directly accessible, and it is difficult to solve the time-varying Riccati equation associated with the finite-horizon setting directly either. The main contribution of the proposed model-free algorithm is to determine the optimal output feedback policies without measurement state in finite-horizon setting. To achieve this goal, we first describe the Q-function caused by finite-horizon problems in the context of state feedback, then we parameterize the Q-functions as input–output vectors functions. Finally, the numerical examples on aircraft dynamics demonstrate the algorithm's efficiency. [ABSTRACT FROM AUTHOR]

Many real world problems are multi-objective. Thus, the need for multi-objective learning and optimization algorithms is inevitable. Although the multi-objective optimization algorithms are well-studied, the multi-objective learning algorithms have attracted less attention. In this paper, a fuzzy multi-objective reinforcement learning algorithm is proposed, and we refer to it as the multi-objective fuzzy Q-learning (MOFQL) algorithm. The algorithm is implemented to solve a bi-objective reach-avoid game. The majority of the multi-objective reinforcement algorithms proposed address solving problems in the discrete state-action domain. However, the MOFQL algorithm can also handle problems in a continuous state-action domain. A fuzzy inference system (FIS) is implemented to estimate the value function for the bi-objective problem. We used a temporal difference (TD) approach to update the fuzzy rules. The proposed method is a multi-policy multi-objective algorithm and can find the non-convex regions of the Pareto front. [ABSTRACT FROM AUTHOR]

Cooperative behavior is essential for conflicts between collective and individual benefits, and evolutionary game theory provides a key framework to solve this problem. Decision-making of human or automata agent occurs not only in a static environment, but also in the dynamic interactive environment. Since the reinforcement learning algorithm is well performed at explaining the problem in regard to state, action, and environment, we propose a game model with individual state transition which is influenced by the self-regarding Q -learning algorithm. In detail, we at the first time investigate a two-state two-action game, where agents can choose either to participate in the network game (i.e., active agent) or to cut off all the links based on its Q -table (i.e., inactive agent), involving in a dynamic interactive environment. Through numerical simulations, it is shown that cooperation can reach the maximal level in the moderate value space of fixed reward obtained by inactive agents. In particular, long-term expectations and large learning rates are more productive in promoting cooperation. Furthermore, when the dynamic interactive environment reaches a stable state, the number of active neighbors of active cooperators is larger than that of active defectors, which is further larger than the number of active neighbors of inactive agents. Finally, we testify the results of theoretical analysis from the perspective of state transition. [ABSTRACT FROM AUTHOR]

In machine learning and pattern recognition tasks, classification performance is often degraded due to the existence of irrelevant and redundant features, especially for high-dimensional data. As a data preprocessing tool, feature selection can improve classification performance while reducing the number of features. Focusing on high-dimensional data, we propose a novel two-stage hybrid feature selection method that combines the maximum information coefficient (MIC) based Q-learning algorithm and the improved particle swarm optimization (PSO) based algorithm, named as MICQ-IPSO. In the first stage, we employed an intelligent feature pre-screening operation to get a rough feature subset, which introduced MIC value as the correlation measure and automated the determination of the screening threshold by Q-learning. In the second stage, we applied an improved PSO-based method to get an optimal feature subset. During this stage, a swarm initialization strategy based on MIC correlation was used to narrow the search range and accelerate swarm convergence. To further enhance the exploitability, a deeper local search operation was performed in the search region. Moreover, a particle reset strategy was adopted to help particles jump out of the local optimal solution. Finally, we evaluated our algorithm against several state-of-the-art feature selection approaches on 17 benchmark datasets. The experimental results demonstrate the effectiveness and competitiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper proposes a novel reinforcement learning-based attitude tracking control strategy for combined spacecraft takeover maneuvers with completely unknown dynamics. One major issue in the context of combined spacecraft attitude takeover control is that the accurate dynamic model is highly nonlinear, complex and costly to identify online, which makes it impractical for control design. To address this issue, we take the advantage of the Q-learning algorithm to acquire the control strategy directly from system input/output measurement data in a model-free manner, and thus the online inertia parameter identification procedure is avoided. More specifically, first, the attitude tracking is formulated as a regulation problem by introducing an argumented system, where the system dynamic model is still required in control design. Then, in order to achieve a model-free control strategy, an online policy-iteration (PI) Q-learning procedure is derived to solve the Bellman optimality equation by utilizing the generated measurement data. In theoretical analysis, it is proved that the iteration sequences of Q value function and control strategy can converge to the optimal ones. In addition, rigorous proof of the stability and monotonicity guarantees of the proposed control strategy are also provided. Furthermore, for the purpose of online implementation, off-policy learning scheme is employed to find the optimal Q value function approximator with neural network structure after data-collection phase. Numerical simulations are exhibited to validate the effectiveness of the proposed strategy. [ABSTRACT FROM AUTHOR]

Cooperative behavior assumes a critical role in resolving conflicts arising between collective and individual interests, while punishment measures serve as a robust deterrent against opportunistic free-riding. Within this context, evolutionary game theory (EGT) emerges as an indispensable paradigm for addressing this multifaceted issue. When it comes to introspection behaviors, reinforcement learning (RL) methods exhibit remarkable capabilities to capture agents' cognitive processes. Nonetheless, previous research has often focused on a static and time-invariant update rule, neglecting the dynamic nature of real-world scenarios where individuals can flexibly transit between strategies in periodic time-dependent patterns. Here, we propose periodic update rules with Q-learning algorithm and game transition model with a punishment mechanism that grants cooperative agents the autonomy to exercise discretion in deciding whether to initiate punishment actions. The agents display dynamic rules periodically through game model transitions, thus ensuring EGT's inherent adaptability. By employing Monte Carlo (MC) simulations, we analyze the emergence of cooperation that underscores the substantial enhancement of cooperative behavior through the proposed periodic update rules with Q-learning algorithm and game transitions in the presence of punishment. Our study highlights the indispensable significance of appropriate periodic intervals for updating rules and determining optimal punishment costs in the game transition model as critical elements for fostering the evolution of cooperation in real-world scenarios. • We pioneeringly propose an improved Q-learning algorithm with periodic update rules, the Fermi and aspiration-driven update rules, to study the evolutionary dynamics of cooperation. • We develop a novel game transition model with a punishment mechanism via the proposed improved Q-learning algorithm to solve the "second-order PDG dilemma". • We experimentally prove the effectiveness of our proposed algorithm and explore how a series of parameters influence the evolution of cooperation through comprehensive Monte Carlo (MC) simulations. [ABSTRACT FROM AUTHOR]

In this article, an accelerated Q-learning algorithm with evolving control is established to solve the optimal tracking control problem. First, an accelerated Q-learning scheme is constructed with an advanced Q-function. By utilizing the advanced Q-function, calculating of the feedforward control input can be avoided and the terminal tracking error can be eliminated. Then, by introducing the relaxation factor, the convergence rate of the iterative Q-function sequence is accelerated significantly, which is a potential way to diminish the computational burden. Furthermore, the convergence, positive definiteness, and stability conditions of the accelerated Q-learning algorithm are analyzed with some preconditions of the relaxation factor. Thus, the developed algorithm can achieve evolving control. Finally, the fantastic performance of the developed algorithm with critic network implementation is verified through two simulation examples. [ABSTRACT FROM AUTHOR]

Recent Electroencephalogram (EEG) analysis in connection with brain disorders has been tremendously benefiting from the (Deep) Neural Network technology in neuroscience research and neuro-engineering practices. However, the performance of existing hand-crafted models, such as the stability, has largely been refrained. This is the case especially in the paradigms that sensitive to the individuality of subjects and the non-stationarity of cognitive dynamics, such as Autism Spectrum Disorder (ASD) evaluation. Aiming at this problem, this study develops a Q-Learning method to enable fast reconstruction of Convolutional Neural Network (CNN) thus to support EEG discrimination adapting to the individuality of subjects under examination. The proposed method first generates a CNN model with the structure and hyper-parameters determined (i.e., Neural Architecture Search) by the customized Q-Learning algorithm, where the CNN model is treated as a discrete system to be optimized. With the sharp shift of subjects, the Q-Learning algorithm reconstructs the CNN model to reach optimization reusing the tacit knowledge learned from the previous trials. A case study has been performed to check the proposed method versus state-of-the-art counterparts based on resting-state EEG collected from 175 ASD-suspicious children with a diverse geological distribution. The observations in the case study indicate that: 1) the method outperforms the counterparts with an individual/sample accuracy of 92.63 % / 83.23 % achieved; 2) the method can quickly reconstruct the CNN model with the group of subjects shifting from one region to another to maintain an encouraging performance while the counterparts without reconstruction may drop by about 12 %. [ABSTRACT FROM AUTHOR]

In this paper, the stability of tracking error dynamics under the data-based discounted iterative Q-learning is investigated. First, a novel performance index with a discount factor is introduced into the iterative Q-learning-based tracking control. Then, considering the approximation errors caused by the Q-function approximator, the finite error bound between the iterative and optimal Q-functions is established. Moreover, based on the new stability analysis, the selection rule of the discount factor is developed, which ensures that the corresponding optimal control policy is admissible. Next, to ensure the stability of the tracking error dynamics under iterative control policies, the stability condition about the approximate Q-function is established. It is guaranteed that iterative control policies derived from the critic network drive the tracking error to zero. Additionally, considering the adopted policy function approximator, the upper bound function of the approximation errors is developed. It is ensured that the trained action network stabilizes the tracking error dynamics. Finally, a simulation example is utilized to implement the data-based discounted iterative Q-learning and verify the present theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, a method of Active Disturbance Rejection Controller (ADRC) is proposed based on Q-learning of Reinforcement Learning (RL) for multi-area interconnected power system. Excessive changes in load can cause instability to the system. Therefore, the ADRC controller is used to keep the load within rated range for its strong anti-interference performance and Q-learning algorithm to select the adaptive parameters of the controller. Finally, through simulation experiments on traditional and deregulated three-area interconnected power system respectively, the effectiveness of the proposed method is proved and the results show that reinforcement learning can indeed be used to solve the problem of controller parameter adjustment. [ABSTRACT FROM AUTHOR]

Recently, deep-learning based lane detection methods effectively boost the development of Advanced Driver Assistance Systems (ADAS) and Self-Driving Systems. However, these methods only detect lane lines with sketchy bounding boxes while ignore the shape of specific curved lanes. To address the above problems, this paper introduces deep reinforcement learning into cursory lane detection models for accurate lane detection and localization. This model consists of two stages, namely the bounding box detector and landmark point localizer. To be specific, a bounding box level convolution neural network lane detector outputs the preliminary location of lanes in the form of bounding boxes. Then, a reinforcement based Deep Q-Learning Localizer (DQLL) accurately localizes the lanes as a group of landmarks to achieve better representation of curved lanes. Moreover, a pixel-level lane detection dataset named NWPU Lanes Dataset is constructed and released. It contains a variety of real traffic scenes and accurate masks of the lane lines. This approach achieves competitive performance in the released dataset and TuSimple Lane dataset. Furthermore, the codes and dataset will be released on https://github.com/tuzixini/DQLL. [ABSTRACT FROM AUTHOR]

• First, we design a LADRC ship course controller to deal with nonlinearity and uncertainty caused by complex sea conditions, such as wind, waves and load. • Second, to achieve better control performance, we propose an adaptive algorithm based on Q-learning. • Furthermore, some desired states are initialized with higher probability during the training in order to train the corresponding Q-value more adequately. This paper concerns with the method of parameters tuning and the capability of active disturbance rejection control (ADRC) to the nonlinear plants. Firstly, an adaptive method of ADRC parameters based on Q-learning is proposed. Besides, to verify the effectiveness of the proposed parameter tuning strategy, the novel method is applied to the ship course control which has multifarious uncertainties due to the disturbance of wind, waves and currents. And then, for better control performance, when the training of Q value-table, the states stochastic initialize of each episode is not equiprobability, different state has different weights. The simulation results of both adaptive ADRC and linear active disturbance rejection control (LADRC) show that the proposed algorithm has the advantages of robustness and higher tracking precision. [ABSTRACT FROM AUTHOR]

In this paper, a novel image-based visual servoing (IBVS) method using Extreme Learning Machine (ELM) and Q -learning is proposed to solve the problems of complex modeling and selection of the servo gain. First, the pseudoinverse of the interaction matrix is approached by ELM which avoids the singularity of the interaction matrix effectively and is robust to interferences such as feature noises and camera calibration errors. Second, a reinforcement learning method, Q -learning, is adopted to adaptively adjust the servo gain in order to improve the convergence speed and stability. Compared with other methods, ELM has better generalization performance, faster operation speed and a unique optimal solution. Also, Q -learning has self-learning ability without experience in advance. The effectiveness of the proposed method is validated by simulations and experiment on a 6-DOF robot with eye-in-hand configuration. [ABSTRACT FROM AUTHOR]

Reinforcement Learning (RL) technologies are powerful to learn how to interact with environments and have been successfully applied to various important applications. Q-learning is one of the most popular methods in RL, which leverages the Bellman equation to update the Q-function. Considering that data collection in RL is both time and cost consuming and Q-learning converges slowly, different kinds of transfer RL algorithms are designed to improve the sample complexity of the new tasks 1 1 In order to avoid confusion, we use "old/new tasks" instead of "source/target tasks" in this paper.. However, most of the previous transfer RL algorithms are similar to the transfer learning methods in deep learning and are heuristic with no theoretical guarantee of the convergence rate. Therefore, it is important for us to clearly understand how and when will transfer learning help RL method and provide the theoretical guarantee for the improvement of the sample complexity. In this paper, we rethink the transfer Rl problems in the RL perspective and propose to transfer the Q-function learned in the old task to the target Q-function in the Q-learning of the new task. We call this new transfer Q-learning method target transfer Q-Learning (abbrev. TTQL). The transfer process is controlled by the error condition which can help to avoid the harm to the new tasks brought by the transferred target. We design the error condition in TTQL as whether the Bellman error of the transferred target Q-function is less than the current Q-function. We show that TTQL with the error condition will achieve a faster convergence rate than Q-learning. Our experiments are consistent with our theoretical results and verify the effectiveness of our proposed target transfer Q-learning method. [ABSTRACT FROM AUTHOR]

• An improved multi-agent system based on Fuzzy ARTMAP (FAM) and Q-learning, i.e., IQ-MACS, is proposed. • A trust measurement using a combination of Q-learning and Bayesian formalism is added at team level. • Two real-world case studies, i.e., human motion detection and motor fault detection, with and without noise are used to evaluate the proposed model. • The outcome indicates that IQMACS is able to produce promising results. The Fuzzy ARTMAP (FAM) network is an online supervised neural network that operates by computing the similarity level between the new sample and those prototype nodes stored in its network against a threshold. In our previous study, we have developed a multi-agent system consisting of an ensemble of FAM networks and Q-learning, known as QMACS, for data classification. In this paper, an Improved QMACS (IQMACS) model with trust measurement using a combination of Q-learning and Bayesian formalism is proposed. A number of benchmark and real-world problems, i.e., motor fault detection and human motion detection, are conducted to evaluate the effectiveness of IQMACS. Statistical features are extracted from real-world case studies and utilized for classification with IQMACS, QMACS, and their constituents. The experimental results indicate that IQMACS produces better classification performance by combining the outcomes of its constituents as compared with those of QMACS and other related methods. [ABSTRACT FROM AUTHOR]

As a pivotal component of microgrid control and optimization, demand response management (DRM) is implemented in bidirectional communication between power suppliers and users to enhance the efficiency of the microgrid, and to balance supply and demand. Unstable communication, especially in wireless networks, will inevitably influence DRM performance. In this paper, a novel cognitive control approach is adopted to relieve the negative impacts on microgrid DRM performance. Unlike previous studies that have tried to resolve the impacts through communication methods, our machine learning method considers both communication quality and lost data estimates. The notion of information gap is introduced to represent information uncertainties caused by unstable communication. Moreover, cognitive control models are formulated in both the user agent and the microgrid control center agent, and a Q-learning algorithm is used as a cognitive learning method to generate different optimal policies for both types of agents based on attainable information. Our simulation results demonstrate the relationship between DRM performance and the unstable wireless environment. Moreover, the effectiveness of our proposed cognitive control approach is verified. [ABSTRACT FROM AUTHOR]

Solving the stochastic linear quadratic (SLQ) optimal control problem generally needs full information about system dynamics. In this paper, a Q-learning iteration algorithm is adopted to solve the control problem for model-free discrete-time systems. Firstly, the condition of the well-posedness for the SLQ problem is given. In order to solve the SLQ problem, the stochastic problem is transformed into the deterministic one. Secondly, in the iteration process of Q-learning algorithm, the H matrix sequence and control gain matrix sequence are obtained without the knowledge of system parameters, and the convergence proof of two sequences is also given. Lastly, two simulation examples are supplied to explain the effectiveness of the Q-learning algorithm. [ABSTRACT FROM AUTHOR]

This paper proposes the design and implementation of a model-free tire slip control for a fast and highly nonlinear Anti-lock Braking System (ABS). A reinforcement Q-learning optimal control approach is inserted in a batch neural fitted scheme using two neural networks to approximate the value function and the controller, respectively. The transition samples required for learning high performance control can be collected by interacting with the process either by online exploiting the current iteration controller (or policy) under an ε-greedy exploration strategy, or by using data collected under any other controller that is capable of ensuring efficient exploration of the action-state space. Both approaches are highlighted in the paper. Fortunately, the ABS process fits this type of learning-by-interaction because it does not need an initial stabilizing controller. The validation case studies conducted on a real laboratory setup reveal that high control system performance can be achieved using the proposed approaches. Insightful comments on the observed control behavior are offered along with performance comparisons with several types of model-based and model-free controllers including relay, model-based optimal PI, an original model-free neural network state-feedback VRFT controller and a model-free neural network adaptive actor-critic one. With the ability to improve control performance starting from different supervisory controllers or to learn high performance controllers from scratch, the proposed Q-learning optimal control approach proves its performance in a wide operating range and is therefore recommended to its industrial application on ABS. [ABSTRACT FROM AUTHOR]

This work describes MPQ-learning, an algorithm that approximates the set of all deterministic non-dominated policies in multi-objective Markov decision problems, where rewards are vectors and each component stands for an objective to maximize. MPQ-learning generalizes directly the ideas of Q-learning to the multi-objective case. It can be applied to non-convex Pareto frontiers and finds both supported and unsupported solutions. We present the results of the application of MPQ-learning to some benchmark problems. The algorithm solves successfully these problems, so showing the feasibility of this approach. We also compare MPQ-learning to a standard linearization procedure that computes only supported solutions and show that in some cases MPQ-learning can be as effective as the scalarization method. [ABSTRACT FROM AUTHOR]

In this work, we present a methodology that enables an agent to make efficient use of its exploratory actions by autonomously identifying possible objectives in its environment and learning them in parallel. The identification of objectives is achieved using an online and unsupervised adaptive clustering algorithm. The identified objectives are learned (at least partially) in parallel using Q − learning. Using a simulated agent and environment, it is shown that the converged or partially converged value function weights resulting from off-policy learning can be used to accumulate knowledge about multiple objectives without any additional exploration. We claim that the proposed approach could be useful in scenarios where the objectives are initially unknown or in real world scenarios where exploration is typically a time and energy intensive process. The implications and possible extensions of this work are also briefly discussed. [ABSTRACT FROM AUTHOR]

Wireless big data significantly challenges the current network management and control architecture, mathematical modeling techniques, and distributed algorithm design, in particular, in the promising cognitive, distributed, and ultra-dense networks. Motivated by the idea of divide-and-conquer, in this article, we first present a multiple cognitive agent-based divide-and-conquer network management and control architecture. Furthermore, a Markovian game-theoretic modeling framework is proposed to model the state big data-based decision-making problem. Then, we investigate various learning methodologies with respect to different kinds of the state information, in particular, we concentrate on the construction of state space, the state transition computation, and the convergence of parallel Q -learning technique. This work provides a suitable network management architecture, an effective modeling tool, and various learning techniques for wireless big data networks. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, a forward-in-time optimal control method for a class of discrete-time nonlinear systems with general multiobjective performance indices is proposed with unknown system dynamics. The proposed approximate dynamic programming (ADP) method aims to find out the increments of both the controls and states instead of computing the controls and states directly. Using the technique of dimension augment, the vector-valued performance indices are transformed into additive quadratic form which satisfies the corresponding discrete-time algebraic Riccati equation (DTARE). Both the action and critic networks can be adaptively tuned by adaptive critic methods without the information of the system model. The convergence property is guaranteed by a rigorous mathematical proof and finally the simulation results show the effectiveness of the method. [Copyright &y& Elsevier]

Abstract: In this paper, a novel framework for automatic generation of fuzzy neural networks (FNNs) termed hierarchically generated fuzzy neural networks (HGFNN) is proposed for realizing machine intelligence. Human intelligence in organizing companies in a civic society has been adopted in this framework. In the HGFNN framework, an FNN is regarded as a company and fuzzy rules are considered as employees of the company. Analogous to the management of a company, three criteria, namely client satisfaction, performance evaluation and cost minimization, have been proposed. Simulation studies on mobile robot control demonstrate that the proposed method is superior to other existing approaches. [Copyright &y& Elsevier]

Abstract: Ensemble algorithms can improve the performance of a given learning algorithm through the combination of multiple base classifiers into an ensemble. In this paper, we attempt to train and combine the base classifiers using an adaptive policy. This policy is learnt through a Q-learning inspired technique. Its effectiveness for an essentially supervised task is demonstrated by experimental results on several UCI benchmark databases. [Copyright &y& Elsevier]