Search

In this paper, a new framework for the resilient control of continuous-time linear systems under denial-of-service (DoS) attacks and system uncertainty is presented. Integrating techniques from reinforcement learning and output regulation theory, it is shown that resilient optimal controllers can be learned directly from real-time state and input data collected from the systems subjected to attacks. Sufficient conditions are given under which the closed-loop system remains stable given any upper bound of DoS attack duration. Simulation results are used to demonstrate the efficacy of the proposed learning-based framework for resilient control under DoS attacks and model uncertainty.

In this paper, we have proposed a resilient reinforcement learning method for discrete-time linear systems with unknown parameters, under denial-of-service (DoS) attacks. The proposed method is based on policy iteration that learns the optimal controller from input-state data amidst DoS attacks. We achieve an upper bound for the DoS duration to ensure closed-loop stability. The resilience of the closed-loop system, when subjected to DoS attacks with the learned controller and an internal model, has been thoroughly examined. The effectiveness of the proposed methodology is demonstrated on an inverted pendulum on a cart.

This paper studies the safety-critical control problem for Euler-Lagrange (EL) systems subject to multiple ball obstacles and velocity constraints in accordance with affordable velocity ranges. A key strategy is to exploit the underlying inner-outer-loop structure for the design of a new cascade controller for the class of EL systems. In particular, the outer-loop controller is developed based on quadratic programming (QP) to avoid ball obstacles and generate velocity reference signals fulfilling the velocity limitation. Taking full advantage of the conservation-of-energy property, a nonlinear velocity-tracking controller is designed to form the inner loop. One major difficulty is caused by the possible non-Lipschitz continuity of the standard QP algorithm when there are multiple constraints. To solve this problem, we propose a refined QP algorithm with the feasible set reshaped by an appropriately chosen positive basis such that the feasibility is retained while the resulting outer-loop controller is locally Lipschitz. It is proved that the constraint-satisfaction problem is solvable as long as the ball obstacles satisfy a mild distance condition. The proposed design is validated by numerical simulation and an experiment based on a $2$-link planar manipulator.

This paper proposes a constructive approach to safety control of nonlinear cascade systems subject to multiple state constraints. New design ingredients include a unified characterization of safety and stability for systematic designs of safety controllers, and a novel technique of reshaping the feasible sets of quadratically constrained quadratic programming induced from safety control. The proposed method guarantees Lipschitz continuity of virtual control laws, enabling a stepwise constructive design. A refined nonlinear small-gain synthesis is employed to address the nonlinear uncertain interconnections between the resulting subsystems corresponding to different virtual control laws, and to guarantee the achievement of the safety control objective. When the safety constraints are removed, the proposed approach coincides with the standard constructive nonlinear control. The proposed safety-control algorithm is experimentally validated in a testbed involving a vertical takeoff and landing (VTOL) vehicle taking off in narrow spaces.

This paper presents a new systematic framework for nonlinear singularly perturbed systems in which state-dependent perturbation functions are used instead of constant perturbation coefficients. Under this framework, general results are obtained for the global robust stability and input-to-state stability of nonlinear singularly perturbed systems. Interestingly, the proposed methodology provides innovative solutions beyond traditional singular perturbation theory for emerging control problems arising from nonlinear integral control, feedback optimization, and formation-based source seeking.

This paper studies the effect of perturbations on the gradient flow of a general nonlinear programming problem, where the perturbation may arise from inaccurate gradient estimation in the setting of data-driven optimization. Under suitable conditions on the objective function, the perturbed gradient flow is shown to be small-disturbance input-to-state stable (ISS), which implies that, in the presence of a small-enough perturbation, the trajectories of the perturbed gradient flow must eventually enter a small neighborhood of the optimum. This work was motivated by the question of robustness of direct methods for the linear quadratic regulator problem, and specifically the analysis of the effect of perturbations caused by gradient estimation or round-off errors in policy optimization. We show small-disturbance ISS for three of the most common optimization algorithms: standard gradient flow, natural gradient flow, and Newton gradient flow., Comment: 20 pages

This paper proposes a novel robust reinforcement learning framework for discrete-time linear systems with model mismatch that may arise from the sim-to-real gap. A key strategy is to invoke advanced techniques from control theory. Using the formulation of the classical risk-sensitive linear quadratic Gaussian control, a dual-loop policy optimization algorithm is proposed to generate a robust optimal controller. The dual-loop policy optimization algorithm is shown to be globally and uniformly convergent, and robust against disturbances during the learning process. This robustness property is called small-disturbance input-to-state stability and guarantees that the proposed policy optimization algorithm converges to a small neighborhood of the optimal controller as long as the disturbance at each learning step is relatively small. In addition, when the system dynamics is unknown, a novel model-free off-policy policy optimization algorithm is proposed. Finally, numerical examples are provided to illustrate the proposed algorithm., Comment: 27 Pages, 13 Figures

This paper studies the control problem for safety-critical multi-agent systems based on quadratic programming (QP). Each controlled agent is modeled as a cascade connection of an integrator and an uncertain nonlinear actuation system. In particular, the integrator represents the position-velocity relation, and the actuation system describes the dynamic response of the actual velocity to the velocity reference signal. The notion of input-to-output stability (IOS) is employed to characterize the essential velocity-tracking capability of the actuation system. The uncertain actuation dynamics may cause infeasibility or discontinuous solutions of QP algorithms for collision avoidance. Also, the interaction between the controlled integrator and the uncertain actuation dynamics may lead to significant robustness issues. By using nonlinear control methods and numerical optimization methods, this paper first contributes a new feasible-set reshaping technique and a refined QP algorithm for feasibility, robustness, and local Lipschitz continuity. Then, we present a nonlinear small-gain analysis to handle the inherent interaction for guaranteed safety of the closed-loop multi-agent system. The proposed methods are illustrated by numerical simulations and a physical experiment.

This paper investigates recursive feasibility, recursive robust stability and near-optimality properties of policy iteration (PI). For this purpose, we consider deterministic nonlinear discrete-time systems whose inputs are generated by PI for undiscounted cost functions. We first assume that PI is recursively feasible, in the sense that the optimization problems solved at each iteration admit a solution. In this case, we provide novel conditions to establish recursive robust stability properties for a general attractor, meaning that the policies generated at each iteration ensure a robust \KL-stability property with respect to a general state measure. We then derive novel explicit bounds on the mismatch between the (suboptimal) value function returned by PI at each iteration and the optimal one. Afterwards, motivated by a counter-example that shows that PI may fail to be recursively feasible, we modify PI so that recursive feasibility is guaranteed a priori under mild conditions. This modified algorithm, called PI+, is shown to preserve the recursive robust stability when the attractor is compact. Additionally, PI+ enjoys the same near-optimality properties as its PI counterpart under the same assumptions. Therefore, PI+ is an attractive tool for generating near-optimal stabilizing control of deterministic discrete-time nonlinear systems., Comment: Submitted for review

This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning and adaptive dynamic programming and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving., Comment: 12 pages, 8 figures

Document-level relation extraction is to extract relation facts from a document consisting of multiple sentences, in which pronoun crossed sentences are a ubiquitous phenomenon against a single sentence. However, most of the previous works focus more on mentions coreference resolution except for pronouns, and rarely pay attention to mention-pronoun coreference and capturing the relations. To represent multi-sentence features by pronouns, we imitate the reading process of humans by leveraging coreference information when dynamically constructing a heterogeneous graph to enhance semantic information. Since the pronoun is notoriously ambiguous in the graph, a mention-pronoun coreference resolution is introduced to calculate the affinity between pronouns and corresponding mentions, and the noise suppression mechanism is proposed to reduce the noise caused by pronouns. Experiments on the public dataset, DocRED, DialogRE and MPDD, show that Coref-aware Doc-level Relation Extraction based on Graph Inference Network outperforms the state-of-the-art.

The potential waning of the vaccination immunity to COVID-19 could pose threats to public health, as it is tenable that the timing of such waning would synchronize with the near-complete restoration of normalcy. Should also testing be relaxed, we might witness a resurgent COVID-19 wave in winter 2021/2022. In response to this risk, an additional vaccine dose, the booster shot, is being administered worldwide. In a projected study with an outlook of six months, we explore the interplay between the rate at which boosters are distributed and the extent to which testing practices are implemented, using a highly granular agent-based model tuned on a medium-sized U.S. town. Theoretical projections indicate that the administration of boosters at the rate at which the vaccine is currently administered could yield a severe resurgence of the pandemic. Projections suggest that the peak levels of mid spring 2021 in the vaccination rate may prevent such a scenario to occur, although exact agreement between observations and projections should not be expected due to continuously evolving nature of the pandemics. Our study highlights the importance of testing, especially to detect asymptomatic individuals in the near future, as the release of the booster reaches full speed., Comment: 56 pages; 15 figures; accepted for publication in Advanced Theory and Simulations

This paper studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness., Comment: 10 pages, 3 figures

Amid the ongoing COVID-19 pandemic, public health authorities and the general population are striving to achieve a balance between safety and normalcy. Ever changing conditions call for the development of theory and simulation tools to finely describe multiple strata of society while supporting the evaluation of "what-if" scenarios. Particularly important is to assess the effectiveness of potential testing approaches and vaccination strategies. Here, an agent-based modeling platform is proposed to simulate the spreading of COVID-19 in small towns and cities, with a single-individual resolution. The platform is validated on real data from New Rochelle, NY -- one of the first outbreaks registered in the United States. Supported by expert knowledge and informed by reported data, the model incorporates detailed elements of the spreading within a statistically realistic population. Along with pertinent functionality such as testing, treatment, and vaccination options, the model accounts for the burden of other illnesses with symptoms similar to COVID-19. Unique to the model is the possibility to explore different testing approaches -- in hospitals or drive-through facilities -- and vaccination strategies that could prioritize vulnerable groups. Decision making by public authorities could benefit from the model, for its fine-grain resolution, open-source nature, and wide range of features., Comment: 44 pages (including 16 of Supplementary Information). Published online in Advanced Theory and Simulations

High-speed signal-free intersections are a novel urban traffic operations enabled by connected and autonomous vehicles. However, the impact of communication latency on intersection performance has not been well understood. In this paper, we consider vehicle coordination at signal-free intersections with latency. We focus on two questions: (i) how to ensure latency-resiliency of the coordination algorithm, and (ii) how latency affects the intersection's capacity. We consider a trajectory-based model with bounded speed uncertainties. Latency leads to uncertain state observation. We propose a piecewise-linear control law that ensures safety (avoidance of interference) as long as the initial condition is safe. We also analytically quantify the throughput that the proposed control can attain in the face of latency.

This paper studies the robustness of reinforcement learning algorithms to errors in the learning process. Specifically, we revisit the benchmark problem of discrete-time linear quadratic regulation (LQR) and study the long-standing open question: Under what conditions is the policy iteration method robustly stable from a dynamical systems perspective? Using advanced stability results in control theory, it is shown that policy iteration for LQR is inherently robust to small errors in the learning process and enjoys small-disturbance input-to-state stability: whenever the error in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded, and, moreover, enter and stay in a small neighbourhood of the optimal LQR solution. As an application, a novel off-policy optimistic least-squares policy iteration for the LQR problem is proposed, when the system dynamics are subjected to additive stochastic disturbances. The proposed new results in robust reinforcement learning are validated by a numerical example., Comment: arXiv admin note: text overlap with arXiv:2005.09528

In this paper, a new reinforcement learning (RL) method known as the method of temporal differential is introduced. Compared to the traditional temporal-difference learning method, it plays a crucial role in developing novel RL techniques for continuous environments. In particular, the continuous-time least squares policy evaluation (CT-LSPE) and the continuous-time temporal-differential (CT-TD) learning methods are developed. Both theoretical and empirical evidences are provided to demonstrate the effectiveness of the proposed temporal-differential learning methodology.

This paper studies the robustness of policy iteration in the context of continuous-time infinite-horizon linear quadratic regulation (LQR) problem. It is shown that Kleinman's policy iteration algorithm is inherently robust to small disturbances and enjoys local input-to-state stability in the sense of Sontag. More precisely, whenever the disturbance-induced input term in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded and enter a small neighborhood of the optimal solution of the LQR problem. Based on this result, an off-policy data-driven policy iteration algorithm for the LQR problem is shown to be robust when the system dynamics are subjected to small additive unknown bounded disturbances. The theoretical results are validated by a numerical example.

This published paper investigates the distributed tracking control problem for a class of Euler-Lagrange multi-agent systems when the agents can only measure the positions. In this case, the lack of the separation principle and the strong nonlinearity in unmeasurable states pose severe technical challenges to global output-feedback control design. To overcome these difficulties, a global nonsingular coordinate transformation matrix in the upper triangular form is firstly proposed such that the nonlinear dynamic model can be partially linearized with respect to the unmeasurable states. And, a new type of velocity observers is designed to estimate the unmeasurable velocities for each system. Then, based on the outputs of the velocity observers, we propose distributed control laws that enable the coordinated tracking control system to achieve uniform global exponential stability (UGES). Both theoretical analysis and numerical simulations are presented to validate the effectiveness of the proposed control scheme. Followed by the original paper, a typo and a mistake is corrected., Comment: The original published paper and its errata are presented

This paper studies the infinite-horizon adaptive optimal control of continuous-time linear periodic (CTLP) systems. A novel value iteration (VI) based off-policy ADP algorithm is proposed for a general class of CTLP systems, so that approximate optimal solutions can be obtained directly from the collected data, without the exact knowledge of system dynamics. Under mild conditions, the proofs on uniform convergence of the proposed algorithm to the optimal solutions are given for both the model-based and model-free cases. The VI-based ADP algorithm is able to find suboptimal controllers without assuming the knowledge of an initial stabilizing controller. Application to the optimal control of a triple inverted pendulum subjected to a periodically varying load demonstrates the feasibility and effectiveness of the proposed method., Comment: 9 pages, 2 figures

This paper presents a new theory, known as robust dynamic pro- gramming, for a class of continuous-time dynamical systems. Different from traditional dynamic programming (DP) methods, this new theory serves as a fundamental tool to analyze the robustness of DP algorithms, and in par- ticular, to develop novel adaptive optimal control and reinforcement learning methods. In order to demonstrate the potential of this new framework, four illustrative applications in the fields of stochastic optimal control and adaptive DP are presented. Three numerical examples arising from both finance and engineering industries are also given, along with several possible extensions of the proposed framework.

This paper presents a novel method of global adaptive dynamic programming (ADP) for the adaptive optimal control of nonlinear polynomial systems. The strategy consists of relaxing the problem of solving the Hamilton-Jacobi-Bellman (HJB) equation to an optimization problem, which is solved via a new policy iteration method. The proposed method distinguishes from previously known nonlinear ADP methods in that the neural network approximation is avoided, giving rise to significant computational improvement. Instead of semiglobally or locally stabilizing, the resultant control policy is globally stabilizing for a general class of nonlinear polynomial systems. Furthermore, in the absence of the a priori knowledge of the system dynamics, an online learning method is devised to implement the proposed policy iteration technique by generalizing the current ADP theory. Finally, three numerical examples are provided to validate the effectiveness of the proposed method., Comment: This is an updated version of the publication "Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems," in IEEE Transactions on Automatic Control, vol. 60, no. 11, pp. 2917-2929, Nov. 2015. Few typos have been fixed in this version

This paper studies the robust optimal control design for uncertain nonlinear systems from a perspective of robust adaptive dynamic programming (robust-ADP). The objective is to fill up a gap in the past literature of ADP where dynamic uncertainties or unmodeled dynamics are not addressed. A key strategy is to integrate tools from modern nonlinear control theory, such as the robust redesign and the backstepping techniques as well as the nonlinear small-gain theorem, with the theory of ADP. The proposed robust-ADP methodology can be viewed as a natural extension of ADP to uncertain nonlinear systems. A practical learning algorithm is developed in this paper, and has been applied to a sensorimotor control problem., Comment: 8 pages, 4 figures

This paper studies the use of vector Lyapunov functions for the design of globally stabilizing feedback laws for nonlinear systems. Recent results on vector Lyapunov functions are utilized. The main result of the paper shows that the existence of a vector control Lyapunov function is a necessary and sufficient condition for the existence of a smooth globally stabilizing feedback. Applications to nonlinear systems are provided: simple and easily checkable sufficient conditions are proposed to guarantee the existence of a smooth globally stabilizing feedback law. The obtained results are applied to the problem of the stabilization of an equilibrium point of a reaction network taking place in a continuous stirred tank reactor., Comment: 25 pages, to be submitted to IEEE Transactions on Automatic Control

A discussion of the robustness properties of the proposed observer with respect to measurement errors is provided for the recently proposed full-order and reduced-order, hybrid, dead-beat observer for a class of nonlinear systems, linear in the unmeasured states., Comment: 9 pages

This paper studies the strong observability property and the reduced-order dead-beat observer design problem for a continuous bioreactor. New relationships between coexistence and strong observability, and checkable sufficient conditions for strong observability, are established for a chemostat with two competing microbial species. Furthermore, the dynamic output feedback stabilization problem is solved for the case of one species.

This paper studies the reduced-order or full-order, dead-beat observer problem for a class of nonlinear systems, linear in the unmeasured states. A novel hybrid observer design strategy is proposed, with the help of the notion of strong observability in finite time. The proposed methodology is applied to a batch reactor, for which a hybrid dead-beat observer is obtained in the absence of the precise measurements of the concentration variables. Moreover, the observer is used for the estimation of the frequency of a sinusoidal signal. The results show that accurate estimations can be provided even if the signal is corrupted by high frequency noise., Comment: 19 pages, 4 figures, submitted to Systems and Control Letters for possible publication

New trajectory-based small-gain results are obtained for nonlinear feedback systems under relaxed assumptions. Specifically, during a transient period, the solutions of the feedback system may not satisfy some key inequalities that previous small-gain results usually utilize to prove stability properties. The results allow the application of the small-gain perspective to various systems which satisfy less demanding stability notions than the Input-to-Output Stability property. The robust global feedback stabilization problem of an uncertain time-delayed chemostat model is solved by means of the trajectory-based small-gain results., Comment: 23 pages, 1 figure. Submitted for possible publication to the International Journal of Robust and Nonlinear Control

This paper develops a novel methodology to study robust stability properties of Nash equilibrium points in dynamic games. Small-gain techniques in modern mathematical control theory are used for the first time to derive conditions guaranteeing uniqueness and global asymptotic stability of Nash equilibrium point for economic models described by functional difference equations. Specification to a Cournot oligopoly game is studied in detail to demonstrate the power of the proposed methodology.

This paper extends the nonlinear ISS small-gain theorem to a large-scale time delay system composed of three or more subsystems. En route to proving this small-gain theorem for systems of differential equations with delays, a small-gain theorem for operators is examined. The result developed for operators allows applications to a wide class of systems, including state space systems with delays., Comment: 6 pages

A new Small-Gain Theorem is presented for general nonlinear control systems. The novelty of this research work is that vector Lyapunov functions and functionals are utilized to derive various input-to-output stability and input-to-state stability results. It is shown that the proposed approach recovers several recent results as special instances and is extendible to several important classes of control systems such as large-scale complex systems, nonlinear sampled-data systems and nonlinear time-delay systems. An application to a biochemical circuit model illustrates the generality and power of the proposed vector small-gain theorem., Comment: Submitted to IEEE Transactions on Automatic Control

Structure and magnetic properties of the compounds $Zn_{1-x}Cu_xCr_2O_4$ (ZCCO) are investigated systematically. A structural phase transition from space-group symmetry $Fd3m$ to $I4_1/amd$ is observed in ZCCO. The critical value of the doping, $x$, appears at $0.58\sim 0.62$ through the appearance of a splitting of diffraction peaks at room temperature. Measurements of dc magnetization, ac susceptibility, memory effect and exchange bias-like (EB-like) effect have been performed to reveal the glassy magnetic behaviors of ZCCO. The system with $x\leqslant 0.50$ is suggested as a spin glass-like (SG-like) of magnetic characterization whereas doping values of $0.58\leqslant x\leqslant 0.90$ defines the system as a $"$cluster glass-like$"$ (CG-like) with unidirectional anisotropy. The Cu content suppresses the geometrical frustration of $ZnCr_2O_4$, which may correlate with the pinning effect of Cu sublattice on Cr sublattice to a preferential direction., Comment: 9 pages, 11 figures

In this work, we propose a methodology for the expression of necessary and sufficient Lyapunov-like conditions for the existence of stabilizing feedback laws. The methodology is an extension of the well-known Control Lyapunov Function (CLF) method and can be applied to very general nonlinear time-varying systems with disturbance and control inputs, including both finite- and infinite-dimensional systems. The generality of the proposed methodology is also reflected upon by the fact that partial stability with respect to output variables is addressed. In addition, it is shown that the generalized CLF method can lead to a novel tool for the explicit design of robust nonlinear controllers for a class of time-delay nonlinear systems with a triangular structure., Comment: 44 pages

In this work stability results for systems described by coupled Retarded Functional Differential Equations (RFDEs) and Functional Difference Equations (FDEs) are presented. The results are based on the observation that the composite system can be regarded as the feedback interconnection of a subsystem described by RFDEs and a subsystem described by FDEs. Recent Small-Gain results and Lyapunov-like characterizations of the Weighted Input-to-Output Stability property for systems described by RFDEs and FDEs are employed. It is shown that the stability results provided in this work can be used to study stability for systems described by neutral functional differential equations and systems described by hyperbolic partial differential equations., Comment: 32 pages, 1 figure

A Small-Gain Theorem, which can be applied to a wide class of systems that includes systems satisfying the weak semigroup property, is presented in the present work. The result generalizes all existing results in the literature and exploits notions of weighted, uniform and non-uniform Input-to-Output Stability (IOS) property. Applications to partial state feedback stabilization problems with sampled-data feedback applied with zero order hold and positive sampling rate, are also presented., Comment: Submitted for possible publication to SIAM Journal on Control and Optimization

In this work characterizations of notions of output stability for uncertain time-varying systems described by retarded functional differential equations are provided. Particularly, characterizations by means of Lyapunov and Razumikhin functions of uniform and non-uniform in time Robust Global Asymptotic Output Stability and Input-to-Output Stability are given. The results of this work have been developed for systems with outputs in abstract normed linear spaces in order to allow outputs with no delay, with discrete or distributed delay or functional outputs with memory., Comment: Submitted for possible publication to the Journal of Differential Equations