Search

This article is to develop an optimized attitude control using reinforcement learning (RL) for a quadrotor unmanned aerial vehicle (QUAV) system. Since this optimized scheme is required to steer both attitude angle position and velocity states to follow the predefined reference signals, it is a very challenging and interesting work. In theory, an optimal control can be found via solving the Hamilton–Jacobi–Bellman (HJB) equation. Nevertheless, the HJB equation associated with the QUAV attitude dynamic equation is hardly solved via analytical methods owing to the complex nonlinearity. To address the problem, the RL strategy of critic-actor architecture using fuzzy logic system approximation is proposed. For smooth manipulating both attitude angle position and velocity states, the optimized control has to contain both tracking error terms corresponding to the angle position and velocity. Thus, if the optimized attitude control learns from the traditional methods, the control algorithm will be very intricate. Since this RL tuning laws are derived from the negative gradient of a simple positive function that is equivalent to the HJB equation, the control algorithm is significantly simplified. Finally, both theory proof and actual experiment demonstrate that this optimized attitude control is competent for fulfilling the control objective.

In this article, the game-based backstepping control method is proposed for the high-order nonlinear multi-agent system with unknown dynamic and input saturation. Reinforcement learning (RL) is employed to get the saddle point solution of the tracking game between each agent and the reference signal for achieving robust control. Specifically, the approximate optimal solution of the established Hamilton–Jacobi–Isaacs (HJI) equation is obtained by policy iteration for each subsystem, and the single network adaptive critic (SNAC) architecture is used to reduce the computational burden. In addition, based on the separation operation of the error term from the derivative of the value function, we achieve the different proportions of the two agents in the game to realize the regulation of the final equilibrium point. Different from the general use of the neural network for system identification, the unknown nonlinear dynamic term is approximated based on the state difference obtained by the command filter. Furthermore, a sufficient condition is established to guarantee that the whole system and each subsystem included are uniformly ultimately bounded. Finally, simulation results are given to show the effectiveness of the proposed method.

In this article, for the single-input–single-output (SISO) nonlinear strict-feedback system, optimized backstepping (OB) control combined with the dynamic surface (DS) technique is developed. OB is to make every subsystem control of backstepping as the optimized one so as to ensure the entire backstepping control being optimized. However, the original design of OB still needs to repeatedly calculate the derivative of virtual controls, as a result, it will inevitably cause the problem of “differential explosion.” In order to alleviate the phenomenon, the OB control is combined with the DS technique. Furthermore, OB control needs to conduct with reinforcement learning (RL) in every backstepping step, hence simplifying the algorithm of RL is very necessary and substantive for achieving the combination. In this work, because the optimized control derives both critic and actor training laws by utilizing a simple positive function instead of the square of approximation of Hamilton–Jacobi–Bellman (HJB) equation, it can obviously simplify the RL algorithm to compare with the traditional optimizing methods. Finally, the feasibility is illustrated via both theory and simulation.

This article addresses a distributed time-varying optimal formation protocol for a class of second-order uncertain nonlinear dynamic multiagent systems (MASs) based on an adaptive neural network (NN) state observer through the backstepping method and simplified reinforcement learning (RL). Each follower agent is subjected to only local information and measurable partial states due to actual sensor limitations. In view of the distributed optimized formation strategic needs, the uncertain nonlinear dynamics and undetectable states may jointly affect the stability of the time-varying cooperative formation control. Furthermore, focusing on Hamilton–Jacobi–Bellman optimization, it is almost incapable of directly dealing with unknown equations. Above uncertainty and immeasurability processed by adaptive state observer and NN simplified RL are further designed to achieve desired second-order formation configuration at the least cost. The optimization protocol can not only solve the undetectable states and realize the prescribed time-varying formation performance on the premise that all the errors are SGUUB, but also prove the stability and update the critics and actors easily. Through the above-mentioned approaches offer an optimal control scheme to address time-varying formation control. Finally, the validity of the theoretical method is proven by the Lyapunov stability theory and digital simulation.

This article is to address the optimized high-order consensus problem of nonlinear canonical dynamic multiagent systems (MASs). Since every agent contains the multiple state variables having differential relation, sliding-mode control (SMC) mechanism is employed for reaching the high-order consensus. Then, reinforcement learning (RL) under identifier–critic–actor architecture is performed for optimizing this consensus control. As a result, RL strategy and SMC mechanism are combined to achieve the optimized high-order consensus control. Compared with the traditional optimal control, on the one hand, the actor and critic RL updating laws are significantly simple and can also make the sufficient training for the adaptive parameters so that the condition of persistence excitation is released; on the other hand, the optimized consensus scheme does not require the complete dynamic acknowledge because the adaptive identifier strategy is integrated into the RL design for compensating the unknown dynamic. As a consequence, this proposed optimized control can smoothly steer the high-order MAS to achieve the consensus of multiple system states. Eventually, both theory and simulation certify feasibility of this control method.

In this article, an optimized leader-following consensus control scheme is proposed for the nonlinear strict-feedback-dynamic multi-agent system by learning from the controlling idea of optimized backstepping technique, which designs the virtual and actual controls of backstepping to be the optimized solution of corresponding subsystems so that the entire backstepping control is optimized. Since this control needs to not only ensure the optimizing system performance but also synchronize the multiple system state variables, it is an interesting and challenging topic. In order to achieve this optimized control, the neural network approximation-based reinforcement learning (RL) is performed under critic-actor architecture. In most of the existing RL-based optimal controls, since both the critic and actor RL updating laws are derived from the negative gradient of square of the Hamilton–Jacobi–Bellman (HJB) equation’s approximation, which contains multiple nonlinear terms, their algorithm are inevitably intricate. However, the proposed optimized control derives the RL updating laws from the negative gradient of a simple positive function, which is correlated with the HJB equation; hence, it can be significantly simple in the algorithm. Meanwhile, it can also release two general conditions, known dynamic and persistence excitation, which are required in most of the RL-based optimal controls. Therefore, the proposed optimized scheme can be a natural selection for the high-order nonlinear multi-agent control. Finally, the effectiveness is demonstrated by both theory and simulation.

In this article, an optimized backstepping (OB) control scheme is proposed for a class of stochastic nonlinear strict-feedback systems with unknown dynamics by using reinforcement learning (RL) strategy of identifier-critic-actor architecture, where the identifier aims to compensate the unknown dynamic, the critic aims to evaluate the control performance and to give the feedback to the actor, and the actor aims to perform the control action. The basic control idea is that all virtual controls and the actual control of backstepping are designed as the optimized solution of corresponding subsystems so that the entire backstepping control is optimized. Different from the deterministic system, stochastic system control needs to consider not only the stochastic disturbance depicted by the Wiener process but also the Hessian term in stability analysis. If the backstepping control is developed on the basis of the published RL optimization methods, it will be difficult to be achieved because, on the one hand, RL of these methods are very complex in the algorithm thanks to their critic and actor updating laws deriving from the negative gradient of the square of approximation of Hamilton–Jacobi–Bellman (HJB) equation; on the other hand, these methods require persistence excitation and known dynamic, where persistence excitation is for training adaptive parameters sufficiently. In this research, both critic and actor updating laws are derived from the negative gradient of a simple positive function, which is yielded on the basis of a partial derivative of the HJB equation. As a result, the RL algorithm can be significantly simplified, meanwhile, two requirements of persistence excitation and known dynamic can be released. Therefore, it can be a natural selection for stochastic optimization control. Finally, from two aspects of theory and simulation, it is demonstrated that the proposed control can arrive at the desired system performance.