Search

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multiagent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this article, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are “independent learners”: they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to the best of our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games. [ABSTRACT FROM AUTHOR]

This article adopts a reinforcement learning (RL) method to solve infinite horizon continuous-time stochastic linear quadratic problems, where the drift and diffusion terms in the dynamics may depend on both the state and control. Based on the Bellman’s dynamic programming principle, we presented an online RL algorithm to attain optimal control with partial system information. This algorithm computes the optimal control, rather than estimates the system coefficients, and solves the related Riccati equation. It only requires local trajectory information, which significantly simplifies the calculation process. We shed light on our theoretical findings using two numerical examples. [ABSTRACT FROM AUTHOR]

This article presents ConVex optimization-based Stochastic steady-state Tracking Error Minimization (CV-STEM), a new state feedback control framework for a class of Itô stochastic nonlinear systems and Lagrangian systems. Its innovation lies in computing the control input by an optimal contraction metric, which greedily minimizes an upper bound of the steady-state mean squared tracking error of the system trajectories. Although the problem of minimizing the bound is nonconvex, its equivalent convex formulation is proposed utilizing SDC parameterizations of the nonlinear system equation. It is shown using stochastic incremental contraction analysis that the CV-STEM provides a sufficient guarantee for exponential boundedness of the error for all time with ${\bf \mathcal {L}_2}$ -robustness properties. For the sake of its sampling-based implementation, we present discrete-time stochastic contraction analysis with respect to a state- and time-dependent metric along with its explicit connection to continuous-time cases. We validate the superiority of the CV-STEM to PID, $\mathcal {H}_\infty$ , and baseline nonlinear controllers for spacecraft attitude control and synchronization problems. [ABSTRACT FROM AUTHOR]

We consider a class of dynamic collective choice models with social interactions, whereby a large number of nonuniform agents have to individually settle on one of multiple discrete alternative choices, with the relevance of their would-be choices continuously impacted by noise and the unfolding group behavior. This class of problems is modeled here as a so-called min-LQG game, i.e., a linear quadratic Gaussian dynamic and noncooperative game, with an additional combinatorial aspect in that it includes a final choice-related minimization in its terminal cost. The presence of this minimization term is key to enforcing some specific discrete choice by each individual agent. The theory of mean field games is invoked to generate a class of decentralized agent feedback control strategies, which are then shown to converge to an exact Nash equilibrium of the game as the number of players increases to infinity. A key building block in our approach is an explicit solution to the problem of computing the best response of a generic agent to some arbitrarily posited smooth mean field trajectory. Ultimately, an agent is shown to face a continuously revised discrete choice problem, where greedy choices dictated by current conditions must be constantly balanced against the risk of the future process noise upsetting the wisdom of such decisions. We show that any Nash equilibrium of the game is defined by an a priori computable probability matrix, which describes the distribution of the players’ choices over the alternatives. The results are illustrated through simulations. [ABSTRACT FROM AUTHOR]

This paper is concerned with controlling poorly known systems, for which only a simplified and rough model is available for control design. There are many systems that cannot be reasonably probed for the sake of identification, yet they are important for areas such as economy, populations, or medicine. The ideas are developed around an alternative way to account for the bare modeling in a stochastic-based setting, and to heighten the control features for such a modified model. The mathematical framework for the optimal control reveals important features such as the raising of a precautionary feedback policy of “keep the action unchanged” (inaction for short), on a certain state-space region. This feature is not seen in the robust approach, but has been pointed out and permeates part of the economics literature. The control problem relies on the viscosity solution for the Hamilton–Jacobi–Bellman equation, and the value of the problem is shown to be convex. When specialized to the quadratic problem with discounted cost, the exact solution inside the inaction region is given by a Lyapunov type of equation, and asymptotically, for large state values, by a Riccati-like equation. This scenario bridges to the stochastic stability analysis for the controlled model. The single control input is developed in full, part analytically, part numerically, for the scalar case, and an approximation is tested for the multidimensional case. The advantage of the precautionary policy is substantial in some situations. [ABSTRACT FROM AUTHOR]

A class of stochastic processes characterized by rapid transitions in their structure is considered and the quickest detection of such transitions is studied in a Bayesian framework. The emphasis is on stochastic processes consisting of a randomly arrived causal pulse (possibly with a set of random parameters such as amplitude and duration) and an additive white Gaussian noise. In this model, the pulse shape and the prior joint density of the arrival time and other random parameters are assumed known. The task of quickest detection in this paper is described mathematically by minimizing the expected detection error. The detection error is represented by a nonlinear function of the distance between the actual transition time and its associated detection time. The assumptions on this function are fairly mild and allow to flexibly design its shape for a desired trade-off between the detection delay and the false alarm rate. Two special cases of such design are well known error measures: mean squared and mean absolute error. The quickest detection problem—a subclass of optimal stopping time problems—is formulated as a stochastic optimal control problem and is resolved using dynamic programming. The optimal detection rule is determined in terms of the solution of an integral equation that cannot be directly solved due to its complexity. This equation is later used to develop a class of suboptimal detection rules and a lower bound on the minimum error. Using this lower bound, it is shown for a numerical example that the suboptimal detector is nearly optimal. [ABSTRACT FROM PUBLISHER]

This technical note presents a continuum approach to a non-Gaussian initial mean consensus problem via Mean Field (MF) stochastic control theory. In this problem formulation: (i) each agent has simple stochastic dynamics with inputs directly controlling its state's rate of change and (ii) each agent seeks to minimize by continuous state feedback its individual discounted cost function involving the mean of the states of all other agents. For this dynamic game problem, a set of coupled deterministic (Hamilton-Jacobi-Bellman and Fokker-Planck-Kolmogorov) equations is derived approximating the stochastic system of agents as the population size goes to infinity. In a finite population system (analogous to the MF LQG framework): (i) the resulting decentralized MF control strategies possess an \epsilon N-Nash equilibrium property where \epsilon N goes to zero as the population size N approaches infinity and (ii) these MF control strategies steer each individual's state toward the initial state population mean which is reached asymptotically as time goes to infinity. Hence, the system with decentralized MF control strategies reaches mean-consensus on the initial state population mean asymptotically as time goes to infinity. [ABSTRACT FROM AUTHOR]

We study large population leader-follower stochastic multi-agent systems where the agents have linear stochastic dynamics and are coupled via their quadratic cost functions. The cost of each leader is based on a trade-off between moving toward a certain reference trajectory which is unknown to the followers and staying near their own centroid. On the other hand, followers react by tracking a convex combination of their own centroid and the centroid of the leaders. We approach this large population dynamic game problem by use of so-called Mean Field (MF) linear-quadratic-Gaussian (LQG) stochastic control theory. In this model, followers are adaptive in the sense that they use a likelihood ratio estimator (on a sample population of the leaders' trajectories) to identify the member of a given finite class of models which is generating the reference trajectory of the leaders. Under appropriate conditions, it is shown that the true reference trajectory model is identified by each follower in finite time with probability one as the leaders' population goes to infinity. Furthermore, we show that the resulting sets of mean field control laws for both leaders and adaptive followers possess an almost sure \varepsilonN-Nash equilibrium property for a system with population N where \varepsilonN goes to zero as N goes to infinity. Numerical experiments are presented illustrating the results. [ABSTRACT FROM AUTHOR]