Search

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. Note to Practitioners— This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

This article provides novel results on the infinite-horizon linear quadratic tracking (LQT) problem for continuous-time systems. In order to fulfill optimality for the tracker design, we introduce a new performance index to reformulate the LQT problem in terms of the minimal polynomial of the exosystem that generates the reference trajectory. We prove that, under routine assumptions, the obtained solution to the formulated LQT problem realizes both closed-loop stability and asymptotic tracking. It is worth pointing out that the proposed procedure does not impose any constraints on the stability of the exosystem. Besides, our method can determine both the optimal control gain and the optimal dynamic compensator, simultaneously. This is different from existing optimal tracker design based on internal model principle, which presets a nonoptimal dynamic compensator. Our theoretical results are corroborated by a simulation example.

This paper proposes a multivariable, adaptive, robust (MAR) control strategy for islanded inverter-based resources (IBRs) operating as grid-forming inverters. The proposed method is employed in the inner control loop of the primary layer in the hierarchical or decentralized structures for the islanded operation of microgrids. The MAR control scheme is responsible for stabilizing IBRs’ output voltage in autonomous operations of microgrids, considering mismatched input voltage disturbances from the grid side and a large amount of system uncertainty. The control methodology introduced in this paper does not rely on the system’s physical parameters, such as microgrid topology, load dynamics, LCL filters, and output connectors. As a result, there is no need to know the nominal values or the bounds of uncertainties in system dynamics. The MAR control method uses online adaptation rules first to identify and then adjust the control parameters of the closed-loop system based on an arbitrary dynamic model. In other words, the MAR method replaces the actual dynamics of IBRs with predetermined dynamics of interest. Simulation results in the MATLAB/Simulink environment confirm the capability of the scheme introduced for the closed-loop stabilization and voltage regulation in the presence of disturbances and a significant amount of uncertainty under various case studies; moreover, comparative simulations by comparing the presented method with other studies using sliding mode control are provided. Finally, experiments verify the effectiveness and practicality of the proposed MAR control scheme. Note to Practitioners—Inverter-based resources are integral parts of current and especially future power and energy systems; with increasing concerns about carbon footprints, the tendency to substitute traditional synchronous generators with inverter-based resources increases. This transition towards the widespread use of power electronics devices needs careful studies regarding the stability and control of power converters. Although existing studies are addressing potential control system challenges, they suffer from complex mathematical computations and the need for the system’s preliminary information. With this in mind, this study proposes a multivariable, adaptive, robust control strategy for the inner voltage control loop of grid-forming inverters. This method utilizes online estimation algorithms to identify inverter-based resources’ parameters and tune control system parameters simultaneously, making it applicable even to cases with slow parameter variations caused by aging or environmental changes. It can compensate for potential voltage disturbances from the grid side and enable the designer to replace undesirable dynamics of inverter-based resource units with arbitrary and stable dynamics of interest. In fact, unlike traditional methods that need control parameters and gains to be tuned to achieve a proper dynamic response, the control system designer can choose reference dynamics and enforce the closed-loop system to imitate the dynamical model selected. This model is usually chosen based on established priorities, such as response time and other transient behaviors. Moreover, this method does not require complex mathematical and algebraic calculations to design and implement. It can be easily applied to inverter-based resource units after selecting the desired reference dynamics, as shown through this study’s experimental result. The above points give this method a competitive edge over the existing algorithms, especially in practical applications.

In this article, a successive approximation learning framework for adaptive optimal control problems, named hybrid iteration (HI), is presented and validated experimentally. The HI strategy outperforms two well-known adaptive dynamic programming strategies, i.e., policy iteration (PI) and value iteration (VI). Using HI, an approximated optimal control policy is learned without the prior knowledge of an admissible control policy required by PI. At the same time, comparing to VI, the HI algorithm converges to the optimal solution with tremendously less learning iterations and CPU-time. Initially, we present the settings of the data-driven HI for continuous-time nonlinear systems and continuous-time linear systems to learn the optimal control policy without any information of the dynamics of the system. Following that, the proposed HI method is implemented on a nonlinear rotary inverted pendulum, and its online learning performance is validated by learning the optimal control policy using online data. The experimental results reveal the efficacy and practicality of the HI method, and demonstrate its superior online learning performance over the traditional PI and VI methods.

This article studies the adaptive optimal output regulation problem for a class of interconnected singularly perturbed systems (SPSs) with unknown dynamics based on reinforcement learning (RL). Taking into account the slow and fast characteristics among system states, the interconnected SPS is decomposed into the slow time-scale dynamics and the fast time-scale dynamics through singular perturbation theory. For the fast time-scale dynamics with interconnections, we devise a decentralized optimal control strategy by selecting appropriate weight matrices in the cost function. For the slow time-scale dynamics with unknown system parameters, an off-policy RL algorithm with convergence guarantee is given to learn the optimal control strategy in terms of measurement data. By combining the slow and fast controllers, we establish the composite decentralized adaptive optimal output regulator, and rigorously analyze the stability and optimality of the closed-loop system. The proposed decomposition design not only bypasses the numerical stiffness but also alleviates the high-dimensionality. The efficacy of the proposed methodology is validated by a load-frequency control application of a two-area power system.