Search

In this work, we consider systems whose components suffer from clock offsets and quantization and study the effect of those on a reinforcement learning (RL) algorithm. Specifically, we consider an off-policy iterative RL algorithm for continuous-time systems, which uses input and state data to approximate the Nash-equilibrium of a zero-sum game. However, the data used by this algorithm are not consistent with one another, in that each of them originates from a slightly different time instant of the past, hence putting the convergence of the algorithm in question. We prove that, given that these timing inconsistencies remain below a certain threshold, the iterative off-policy RL algorithm will still converge epsilon-closely to the desired Nash policy. However, this result is conditional to a certain Lipschitz continuity and differentiability condition on the input-state data collected, which is indispensable in the presence of clock offsets. A similar result is also derived when quantization of the measured state is considered. Finally, unlike prior work, we provide a sufficiently rich data condition for the execution of the iterative RL algorithm, which can be verified a priori across all iteration indices. Simulations are performed, which verify and clarify theoretical findings. [ABSTRACT FROM AUTHOR]

The infinite-horizon zero-sum game of a linear system can be resorted to solve a Game algebraic Riccati equation (GARE) with indefinite quadratic term. Double-loop policy iteration algorithm is often used to find the solution of such GARE, but its calculation is usually time-consuming. In this work, we propose a novel model-based single-loop policy iteration algorithm to solve GARE and the convergence of the algorithm is guaranteed by the boundness of the iterative sequence and the comparison result. Furthermore, we devise a data-driven single-loop policy iteration algorithm for solving linear zero-sum games, without requiring the knowledge of system dynamics. Compared to the existing Newton's single-loop methods, the initialization of our algorithms is significantly relaxed and easier to implement. Two numerical examples are included to illustrate the proposed algorithms. [ABSTRACT FROM AUTHOR]

In an adversarial environment, a hostile player performing a task may behave like a non-hostile one in order not to reveal its identity to an opponent. To model such a scenario, we define identity concealment games: zero-sum stochastic reachability games with a zero-sum objective of identity concealment. To measure the identity concealment of the player, we introduce the notion of an average player. The average player's policy represents the expected behavior of a non-hostile player. We show that there exists an equilibrium policy pair for every identity concealment game and give the optimality equations to synthesize an equilibrium policy pair. If the player's opponent follows a non-equilibrium policy, the player can hide its identity better. For this reason, we study how the hostile player may learn the opponent's policy. Since learning via exploration policies would quickly reveal the hostile player's identity to the opponent, we consider the problem of learning a near-optimal policy for the hostile player using the game runs collected under the average player's policy. Consequently, we propose an algorithm that provably learns a near-optimal policy and give an upper bound on the number of sample runs to be collected. [ABSTRACT FROM AUTHOR]

In this paper, we consider a security-aware and learning-based actuator selection formulation for cyber–physical systems. A set of actuators is chosen to be used by the system so that a metric of controllability and actuation attack resilience is maximized, given only partial knowledge of the physics of the system. Such a metric is related to the trace of a game theoretic Gramian, which is obtained by solving a zero-sum game between a defender who acts as a minimizer and wants to regulate the system, and an attacker who acts as a maximizer with an eventual goal to disrupt the regulation. To account for the lack of full knowledge of the system dynamics, we design an estimator that can learn the trace of the proposed game theoretic Gramian online, by utilizing data obtained across the system's trajectories. Subsequently, we propose a scheduling algorithm that utilizes the estimator to select actuators for the system in an online manner. In fact, we prove that the set of actuators that optimize the game-based metric will be learnt and scheduled permanently in finite time. Finally, we show that the learning-based actuator placement procedure can be utilized, in its steady state, to detect a large class of actuation attacks in a partially model-free manner. We carry out a simulation on an aircraft model to show the efficacy of our framework. [ABSTRACT FROM AUTHOR]

This work presents an output-feedback policy learning algorithm underlining input–output system data for distributed robust optimal synchronization of heterogeneous multi-agent systems. The output-feedback synchronization problem in the context of this work is formulated via robust output regulation and reinforcement learning modeling the interactions among agents by a zero-sum game. The proposed learning and control structure only requires the local system data for each agent and distributed output data among communicating neighbors. We utilize system-level synchysis for the continuous-time state reconstruction for the distributed learning with convergence and stability proofs under the proposed output-feedback policy for solving the zero-sum game. We further show that policy learning is assured under the proposed data criteria relating to input–output data only rather than any inter-immediate gains from policy iterations. Based on the cooperative robust output regulation, this work gains robustness after the learning is complete and establishes an output data-driven distributed optimal robust synchronization without knowing accurate system dynamics. A numerical example shows the effectiveness of the proposed learning algorithm. [ABSTRACT FROM AUTHOR]

The paper introduces a class of zero-sum games between the adversary and controller as a scenario for a ‘denial of service’ in a networked control system. The communication link is modelled as a set of transmission regimes controlled by a strategic jammer whose intention is to wage an attack on the plant by choosing a most damaging regime-switching strategy. We demonstrate that even in the one-step case, the introduced games admit a saddle-point equilibrium, at which the jammer’s optimal policy is to randomize in a region of the plant’s state space, thus requiring the controller to undertake a nontrivial response which is different from what one would expect in a standard stochastic control problem over a packet dropping link. The paper derives conditions for the introduced games to have such a saddle-point equilibrium. Furthermore, we show that in more general multi-stage games, these conditions provide ‘greedy’ jamming strategies for the adversary. [ABSTRACT FROM AUTHOR]

Stochastic optimal control usually requires an explicit dynamical model with probability distributions, which are difficult to obtain in practice. In this work, we consider the linear quadratic regulator (LQR) problem of unknown linear systems and adopt a Wasserstein penalty to address the distribution uncertainty of additive stochastic disturbances. By constructing an equivalent deterministic game of the penalized LQR problem, we propose a Q-learning method with convergence guarantees to learn an optimal minimax controller. [ABSTRACT FROM AUTHOR]

This paper proposes a novel control approach to solve the cooperative H ∞ output regulation problem for linear continuous-time multi-agent systems (MASs). Different from existing solutions to cooperative output regulation problems, a distributed feedforward-feedback controller is developed to achieve asymptotic tracking and reject both modeled and unmodeled disturbances. The feedforward control policy is computed via solving regulator equations, and the optimal feedback control policy is obtained through handling a zero-sum game. Instead of relying on the knowledge of system matrices in the state equations of the followers' dynamics and initial stabilizing feedback control gains, a value iteration (VI) algorithm is proposed to learn the optimal feedback control gain and feedforward control gain using online data. To the best of our knowledge, this paper is the first to show that the proposed VI algorithm can approximate the solution to continuous-time game algebraic Riccati equations with guaranteed convergence. Finally, the numerical analysis is provided to show the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Abstract: This paper addresses the solution of large zero-sum matrix games using randomized methods. We formalize a procedure, termed as the sampled security policy (SSP) algorithm, by which a player can compute policies that, with a high confidence, are security policies against an adversary using randomized methods to explore the possible outcomes of the game. The SSP algorithm essentially consists of solving a stochastically sampled subgame that is much smaller than the original game. We also propose a randomized algorithm, termed as the sampled security value (SSV) algorithm, which computes a high-confidence security-level (i.e., worst-case outcome) for a given policy, which may or may not have been obtained using the SSP algorithm. For both the SSP and the SSV algorithms we provide results to determine how many samples are needed to guarantee a desired level of confidence. We start by providing results when the two players sample policies with the same distribution and subsequently extend these results to the case of mismatched distributions. We demonstrate the usefulness of these results in a hide-and-seek game that exhibits exponential complexity. [Copyright &y& Elsevier]