Search

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multiagent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this article, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are “independent learners”: they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to the best of our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games. [ABSTRACT FROM AUTHOR]

This article adopts a reinforcement learning (RL) method to solve infinite horizon continuous-time stochastic linear quadratic problems, where the drift and diffusion terms in the dynamics may depend on both the state and control. Based on the Bellman’s dynamic programming principle, we presented an online RL algorithm to attain optimal control with partial system information. This algorithm computes the optimal control, rather than estimates the system coefficients, and solves the related Riccati equation. It only requires local trajectory information, which significantly simplifies the calculation process. We shed light on our theoretical findings using two numerical examples. [ABSTRACT FROM AUTHOR]

This article presents ConVex optimization-based Stochastic steady-state Tracking Error Minimization (CV-STEM), a new state feedback control framework for a class of Itô stochastic nonlinear systems and Lagrangian systems. Its innovation lies in computing the control input by an optimal contraction metric, which greedily minimizes an upper bound of the steady-state mean squared tracking error of the system trajectories. Although the problem of minimizing the bound is nonconvex, its equivalent convex formulation is proposed utilizing SDC parameterizations of the nonlinear system equation. It is shown using stochastic incremental contraction analysis that the CV-STEM provides a sufficient guarantee for exponential boundedness of the error for all time with ${\bf \mathcal {L}_2}$ -robustness properties. For the sake of its sampling-based implementation, we present discrete-time stochastic contraction analysis with respect to a state- and time-dependent metric along with its explicit connection to continuous-time cases. We validate the superiority of the CV-STEM to PID, $\mathcal {H}_\infty$ , and baseline nonlinear controllers for spacecraft attitude control and synchronization problems. [ABSTRACT FROM AUTHOR]

We consider a class of dynamic collective choice models with social interactions, whereby a large number of nonuniform agents have to individually settle on one of multiple discrete alternative choices, with the relevance of their would-be choices continuously impacted by noise and the unfolding group behavior. This class of problems is modeled here as a so-called min-LQG game, i.e., a linear quadratic Gaussian dynamic and noncooperative game, with an additional combinatorial aspect in that it includes a final choice-related minimization in its terminal cost. The presence of this minimization term is key to enforcing some specific discrete choice by each individual agent. The theory of mean field games is invoked to generate a class of decentralized agent feedback control strategies, which are then shown to converge to an exact Nash equilibrium of the game as the number of players increases to infinity. A key building block in our approach is an explicit solution to the problem of computing the best response of a generic agent to some arbitrarily posited smooth mean field trajectory. Ultimately, an agent is shown to face a continuously revised discrete choice problem, where greedy choices dictated by current conditions must be constantly balanced against the risk of the future process noise upsetting the wisdom of such decisions. We show that any Nash equilibrium of the game is defined by an a priori computable probability matrix, which describes the distribution of the players’ choices over the alternatives. The results are illustrated through simulations. [ABSTRACT FROM AUTHOR]

A novel risk-sensitive (RS) control law is proposed for linear systems with stochastic system parameters. RS control laws are efficient means of handling risk in various failures in control caused by stochastic disturbances. However, stochastic parameters invoke problems, resulting controllers may become nonlinear in the state variable and incompatible with linear systems, cannot be obtained in an exact sense, and are defined only on a bounded state region. To solve these problems, this paper presents a risk-sensitive linear (RSL) control method. The important idea is that a standard RS-type cost function is converted to an expectation of a weighted cost function such that the resulting optimal controller is linear in the state. The weight is designed such that the weighted cost function preserves the characteristics of the original RS control. The designed weight allows one to derive the proposed RSL controller whose exact solution is obtained as a linear feedback law for all states. Furthermore, the RSL control law over an infinite horizon guarantees stochastic stability of the feedback system with the control law. The effectiveness of the proposed RSL control law is demonstrated by a numerical simulation. [ABSTRACT FROM AUTHOR]

The topic of this paper is model predictive control (MPC) for max-plus linear systems with stochastic uncertainties the distribution of which is supposed to be known. We consider linear constraints on the inputs and the outputs. Due to the uncertainties, these linear constraints are formulated as probabilistic or chance constraints, i.e., the constraints are required to be satisfied with a predefined probability level. The proposed chance constraints can be equivalently rewritten into a max-affine (i.e., the maximum of affine terms) form if the linear constraints are monotonically nondecreasing as a function of the outputs. Based on the resulting max-affine form, two methods are developed for solving the chance-constrained MPC problem for stochastic max-plus linear systems. Method 1 uses Boole's inequality to convert the multivariate chance constraint into univariate chance constraints for which the probability can be computed more efficiently. Method 2 employs Chebyshev's inequality and transforms the chance constraint into linear constraints on the inputs. The simulation results for a production system example show that the two proposed methods are faster than the Monte Carlo simulation method and yield lower closed-loop costs than the nominal MPC method. [ABSTRACT FROM AUTHOR]

This paper is concerned with controlling poorly known systems, for which only a simplified and rough model is available for control design. There are many systems that cannot be reasonably probed for the sake of identification, yet they are important for areas such as economy, populations, or medicine. The ideas are developed around an alternative way to account for the bare modeling in a stochastic-based setting, and to heighten the control features for such a modified model. The mathematical framework for the optimal control reveals important features such as the raising of a precautionary feedback policy of “keep the action unchanged” (inaction for short), on a certain state-space region. This feature is not seen in the robust approach, but has been pointed out and permeates part of the economics literature. The control problem relies on the viscosity solution for the Hamilton–Jacobi–Bellman equation, and the value of the problem is shown to be convex. When specialized to the quadratic problem with discounted cost, the exact solution inside the inaction region is given by a Lyapunov type of equation, and asymptotically, for large state values, by a Riccati-like equation. This scenario bridges to the stochastic stability analysis for the controlled model. The single control input is developed in full, part analytically, part numerically, for the scalar case, and an approximation is tested for the multidimensional case. The advantage of the precautionary policy is substantial in some situations. [ABSTRACT FROM AUTHOR]

Sequential rate-distortion (SRD) theory provides a framework for studying the fundamental trade-off between data-rate and data-quality in real-time communication systems. In this paper, we consider the SRD problem for multi-dimensional time-varying Gauss-Markov processes under mean-square distortion criteria. We first revisit the sensor-estimator separation principle, which asserts that considered SRD problem is equivalent to a joint sensor and estimator design problem in which data-rate of the sensor output is minimized while the estimator's performance satisfies the distortion criteria. We then show that the optimal joint design can be performed by semidefinite programming. A semidefinite representation of the corresponding SRD function is obtained. Implications of the obtained result in the context of zero-delay source coding theory and applications to networked control theory are also discussed. [ABSTRACT FROM AUTHOR]

In this paper, an offline approximate dynamic programming approach using neural networks is proposed for solving a class of finite horizon stochastic optimal control problems. There are two approaches available in the literature, one based on stochastic maximum principle (SMP) formalism and the other based on solving the stochastic Hamilton–Jacobi–Bellman (HJB) equation. However, in the presence of noise, the SMP formalism becomes complex and results in having to solve a couple of backward stochastic differential equations. Hence, current solution methodologies typically ignore the noise effect. On the other hand, the inclusion of noise in the HJB framework is very straightforward. Furthermore, the stochastic HJB equation of a control-affine nonlinear stochastic system with a quadratic control cost function and an arbitrary state cost function can be formulated as a path integral (PI) problem. However, due to curse of dimensionality, it might not be possible to utilize the PI formulation for obtaining comprehensive solutions over the entire operating domain. A neural network structure called the adaptive critic design paradigm is used to effectively handle this difficulty. In this paper, a novel adaptive critic approach using the PI formulation is proposed for solving stochastic optimal control problems. The potential of the algorithm is demonstrated through simulation results from a couple of benchmark problems. [ABSTRACT FROM PUBLISHER]

This paper examines stochastic optimal control problems in which the state is perfectly known, but the controller's measure of time is a stochastic process derived from a strictly increasing Lévy process. We provide dynamic programming results for continuous-time finite-horizon control and specialize these results to solve a noisy-time variant of the linear quadratic regulator problem and a portfolio optimization problem with random trade activity rates. For the linear quadratic case, the optimal controller is linear and can be computed from a generalization of the classical Riccati differential equation. [ABSTRACT FROM PUBLISHER]

• A stochastic optimal control is proposed to stabilize electrostatic MEMS resonators with noise disturbance. • The controller can be used to suppress the noise-activated pull-in instability. • The controller shows superior performance on rejecting the deterministic pull-in instability. • Time delay existing in the optimal control has a periodic impact on the control effectiveness. Noise-induced motions are a significant source of uncertainty in the response of electrostatic MEMS where electrical and mechanical sources contribute to noise and can result in sudden and dramatic loss of stability. In this manuscript, we present an analysis for the stabilization of an electrostatically actuated MEMS resonator in the presence of noise processes using a stochastic optimal control scheme. A stochastic lumped-mass model that accounts for the uncertainty in mass, mechanical restoring force, bias voltage, and AC voltage amplitude of the resonator is presented. The It o ^ equations, describing the averaged modulations of the resonator's total energy and phase difference, are obtained via the stochastic averaging of energy envelope. The stochastic optimal control law aiming at minimization the pull-in probability of the resonator is determined via the stochastic dynamic programming equations associated with the It o ^ equations. We show that the resulting stochastic optimal feedback control, with a careful selection of its control gain, can be used to suppress the noise-activated pull-in instability of the disturbed resonator. The impact of the time delay inherent in the feedback control input on the control effectiveness is investigated. In addition, the control scheme also shows superior performance in counteracting the deterministic pull-in instability of the resonator operating in the dynamic pull-in frequency band. Good agreement between the theoretical results and numerical simulation is demonstrated. [ABSTRACT FROM AUTHOR]

A class of stochastic processes characterized by rapid transitions in their structure is considered and the quickest detection of such transitions is studied in a Bayesian framework. The emphasis is on stochastic processes consisting of a randomly arrived causal pulse (possibly with a set of random parameters such as amplitude and duration) and an additive white Gaussian noise. In this model, the pulse shape and the prior joint density of the arrival time and other random parameters are assumed known. The task of quickest detection in this paper is described mathematically by minimizing the expected detection error. The detection error is represented by a nonlinear function of the distance between the actual transition time and its associated detection time. The assumptions on this function are fairly mild and allow to flexibly design its shape for a desired trade-off between the detection delay and the false alarm rate. Two special cases of such design are well known error measures: mean squared and mean absolute error. The quickest detection problem—a subclass of optimal stopping time problems—is formulated as a stochastic optimal control problem and is resolved using dynamic programming. The optimal detection rule is determined in terms of the solution of an integral equation that cannot be directly solved due to its complexity. This equation is later used to develop a class of suboptimal detection rules and a lower bound on the minimum error. Using this lower bound, it is shown for a numerical example that the suboptimal detector is nearly optimal. [ABSTRACT FROM PUBLISHER]

Optimal sensor scheduling with applications to networked estimation and control systems is considered. We model sensor measurement and transmission instances using jumps between states of a continuous-time Markov chain. We introduce a cost function for this Markov chain as the summation of terms depending on the average sampling frequencies of the subsystems and the effort needed for changing the parameters of the underlying Markov chain. By minimizing this cost function through extending Brockett's recent approach to optimal control of Markov chains, we extract an optimal scheduling policy to fairly allocate the network resources among the control loops. We study the statistical properties of this scheduling policy in order to compute upper bounds for the closed-loop performance of the networked system, where several decoupled scalar subsystems are connected to their corresponding estimator or controller through a shared communication medium. We generalize the estimation results to observable subsystems of arbitrary order. Finally, we illustrate the developed results numerically on a networked system composed of several decoupled water tanks. [ABSTRACT FROM PUBLISHER]

We derive an explicit characterization of the stability region of stationary multi-queue multi-server (MQMS) queueing system by means of a finite set of linear inequalities. More specifically, we explicitly determine the coefficients of the linear inequalities describing the facet-defining hyperplanes of the stability region polytope. Such a characterization is useful for performance evaluation of certain scheduling algorithms such as maximum weight (MW) policy. Our results can be used for studying the asymptotic behavior of the MW policy and computing bounds for the average queueing delay, as well as limiting moments of the queue sizes in heavy-traffic regime. Furthermore, it may be directly applied as the constraint set of network stochastic optimization problems to provide an offline computational solution for such problems. Finally, we use our methodology to characterize the stability region of a fluid model MQMS system which is described by an infinite number of linear inequalities. For such a model, we present an example and show that depending on the channel distribution, the stability region can be instead characterized by a finite set of non-linear inequalities. [ABSTRACT FROM AUTHOR]

This technical note presents a continuum approach to a non-Gaussian initial mean consensus problem via Mean Field (MF) stochastic control theory. In this problem formulation: (i) each agent has simple stochastic dynamics with inputs directly controlling its state's rate of change and (ii) each agent seeks to minimize by continuous state feedback its individual discounted cost function involving the mean of the states of all other agents. For this dynamic game problem, a set of coupled deterministic (Hamilton-Jacobi-Bellman and Fokker-Planck-Kolmogorov) equations is derived approximating the stochastic system of agents as the population size goes to infinity. In a finite population system (analogous to the MF LQG framework): (i) the resulting decentralized MF control strategies possess an \epsilon N-Nash equilibrium property where \epsilon N goes to zero as the population size N approaches infinity and (ii) these MF control strategies steer each individual's state toward the initial state population mean which is reached asymptotically as time goes to infinity. Hence, the system with decentralized MF control strategies reaches mean-consensus on the initial state population mean asymptotically as time goes to infinity. [ABSTRACT FROM AUTHOR]

Subsystems that are coupled due to dynamics and costs arise naturally in various communication applications. In many such applications the control actions are shared between different control stations giving rise to a control sharing information structure. Previous studies of control-sharing have concentrated on the linear quadratic Gaussian setup and a solution approach tailored to continuous valued control actions. In this paper a three step solution approach for finite valued control actions is presented. In the first step, a person-by-person approach is used to identify redundant data or a sufficient statistic for local information at each control station. In the second step, the common-information based approach of Nayyar (2013) is used to find a sufficient statistic for the common information shared between all control stations and to obtain a dynamic programming decomposition. In the third step, the specifics of the model are used to simplify the sufficient statistic and the dynamic program. [ABSTRACT FROM AUTHOR]

Digital control design is commonly constrained to time-triggered control systems with periodic sampling. The emergence of more and more complex and distributed systems urges the development of advanced triggering schemes that utilize communication, computation, and energy resources more efficiently. This technical note addresses the question whether certainty equivalence is optimal for an event-triggered control system with resource constraints. The problem setting is an extension of the stochastic linear quadratic system framework, where the joint design of the control law and the event-triggering law minimizing a common objective is considered. Three differing variants are studied that reflect the resource constraints: a penalty term to acquire the resource, a limitation on the number of resource acquisitions, and a constraint on the average number of resource acquisitions. By reformulating the underlying optimization problem, a characterization of the optimal control law is possible. This characterization shows that the certainty equivalence controller is optimal for all three optimization problems. [ABSTRACT FROM AUTHOR]

We study large population leader-follower stochastic multi-agent systems where the agents have linear stochastic dynamics and are coupled via their quadratic cost functions. The cost of each leader is based on a trade-off between moving toward a certain reference trajectory which is unknown to the followers and staying near their own centroid. On the other hand, followers react by tracking a convex combination of their own centroid and the centroid of the leaders. We approach this large population dynamic game problem by use of so-called Mean Field (MF) linear-quadratic-Gaussian (LQG) stochastic control theory. In this model, followers are adaptive in the sense that they use a likelihood ratio estimator (on a sample population of the leaders' trajectories) to identify the member of a given finite class of models which is generating the reference trajectory of the leaders. Under appropriate conditions, it is shown that the true reference trajectory model is identified by each follower in finite time with probability one as the leaders' population goes to infinity. Furthermore, we show that the resulting sets of mean field control laws for both leaders and adaptive followers possess an almost sure \varepsilonN-Nash equilibrium property for a system with population N where \varepsilonN goes to zero as N goes to infinity. Numerical experiments are presented illustrating the results. [ABSTRACT FROM AUTHOR]

This note investigates the optimal linear quadratic control problem in the discrete-time framework, for stochastic systems affected by disturbances generated by a nonlinear stochastic exosystem. The application of the maximum principle to nonlinear optimal control problems does not admit, in general, implementable solutions. Therefore, it is worthwhile to look for finite-dimensional approximation schemes. The approach followed in this note is based on the -degree Carleman approximation of a stochastic nonlinear system applied to the exosystem and provides a real-time algorithm to design an implementable control law. Simulations support theoretical results and show the improvements when the approximation index ν is increased. [ABSTRACT FROM AUTHOR]