Search

In this paper, we construct a stochastic SAIR (Susceptible–Asymptomatic–Infected–Removed) epidemic model to study the dynamic and control strategy of COVID-19. The existence and uniqueness of the global positive solution are obtained by using the Lyapunov method. We prove the necessary conditions for the existence of extinction and ergodic stationary distribution by defining two new thresholds, respectively. Through the stochastic control theory, the optimal control strategy is obtained. Numerical simulations show the validity of stationary distribution and optimal control. The parameters of the model are estimated by a set of real COVID-19 data. And, the sensitivity of all parameters shows that decreasing physical interaction and screening the asymptomatic as swiftly as possible can prevent the wide spread of the virus in communities. Finally, we also display the trend of the epidemic without control strategies. [ABSTRACT FROM AUTHOR]

Sample complexity bounds are a common performance metric in the reinforcement learning literature. In the discounted cost, infinite horizon setting, all of the known bounds can be arbitrarily large, as the discount factor approaches unity. These results seem to imply that a very large number of samples is required to achieve an epsilon-optimal policy. The objective of the present work is to introduce a new class of algorithms that have sample complexity uniformly bounded over all discount factors. One may argue that this is impossible, due to a recent min–max lower bound. The explanation is that these prior bounds concern value function approximation and not policy approximation. We show that the asymptotic covariance of the tabular Q-learning algorithm with an optimized step-size sequence is a quadratic function of a factor that goes to infinity, as discount factor approaches 1; an essentially known result. The new relative Q-learning algorithm proposed here is shown to have asymptotic covariance that is uniformly bounded over all discount factors. [ABSTRACT FROM AUTHOR]

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multiagent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this article, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are “independent learners”: they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to the best of our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games. [ABSTRACT FROM AUTHOR]

This article adopts a reinforcement learning (RL) method to solve infinite horizon continuous-time stochastic linear quadratic problems, where the drift and diffusion terms in the dynamics may depend on both the state and control. Based on the Bellman’s dynamic programming principle, we presented an online RL algorithm to attain optimal control with partial system information. This algorithm computes the optimal control, rather than estimates the system coefficients, and solves the related Riccati equation. It only requires local trajectory information, which significantly simplifies the calculation process. We shed light on our theoretical findings using two numerical examples. [ABSTRACT FROM AUTHOR]

An unaware susceptible-aware susceptible-vaccinated-infected-recovered (S u S a V IR) epidemic disease model has been developed and extensively examined to better understand the intricate dynamics of the disease transmission with saturated recovery function. A non-linear, monotonically increasing awareness function has been incorporated and its application is contingent upon the number of infected individuals. The spreading threshold R 0 and its sensitivity indices are computed to determine both the prevalence and the potential decline of the spread of disease. Contour plots have been generated to explain how alterations in key parameters affect the evolution of infected individuals. In deterministic model, the condition R 0 < 1 ensures the global stability of a disease-free state while R 0 > 1 indicates the presence of a globally stable prevailing state. A numerical approach has been employed to pinpoint the stability switches for R 0 which indicates transcritical bifurcation. Subsequently, a corresponding stochastic model is developed to investigate the impact of random external factors represented as white noise. In addition, the global existence and uniqueness of solutions, and the asymptotic behavior of solutions in the vicinity of steady-states are exhibited in the stochastic model. In the stochastic model, the extinction threshold R e s is consistently below R 0 due to noise intensity, leading to quicker disease extinction. In order to elucidate the influence of parameters and noise intensities on persistence threshold R p s and extinction threshold R e s , a comprehensive sensitivity analysis and the variation of each noise intensity have been performed. This study shows that treatment can be used as an effective tool for controlling the spread of disease by lowering R 0 and R p s. It has been empirically observed that high environmental fluctuations can serve as a suppressive factor for disease spread in stochastic model with R 0 , R p s > 1 and the higher noise intensities lead to a faster disappearance of the disease. It is noticed that randomness in susceptible and infected individuals can exert a more significant influence on disease mitigation. The spread of infection exhibits a noteworthy dependence on environmental fluctuations. Additionally, to combat and mitigate the spread of disease, we have executed stochastic optimal control measures. Consequently, the control strategy has emerged as a potent tools for effectively managing the propagation of disease. • Nonlinear and non-decreasing awareness programs linked to infected individuals. • Behavior of trajectories in both systems for R 0 < 1 and R 0 > 1 and switch stability. • Stochastic model reveals the effect of noise intensities for disease extinction and persistence. • Stochastic optimal control for the management of disease spread. [ABSTRACT FROM AUTHOR]

We analyze a tree search problem with an underlying Markov decision process, in which the goal is to identify the best action at the root that achieves the highest cumulative reward. We present a new tree policy that optimally allocates a limited computing budget to maximize a lower bound on the probability of correctly selecting the best action at each node. Compared to widely used upper confidence bound (UCB) tree policies, the new tree policy presents a more balanced approach to manage the exploration and exploitation tradeoff when the sampling budget is limited. Furthermore, UCB assumes that the support of reward distribution is known, whereas our algorithm relaxes this assumption. Numerical experiments demonstrate the efficiency of our algorithm in selecting the best action at the root. [ABSTRACT FROM AUTHOR]

In this article, we study the problem of information sharing among rational self-interested agents as a dynamic game of asymmetric information. We assume that the agents imperfectly observe a Markov chain, and they are called to decide whether they will share their noisy observations or not at each time instant. We utilize the notion of conditional mutual information to evaluate the information being shared among the agents. The challenges that arise due to the interdependence of agents’ information structure and decision making are exhibited. For the finite horizon game, we prove that agents do not have incentive to share information. In contrast, we show that cooperation can be sustained in the infinite-horizon case by devising appropriate punishment strategies, which are defined over the agents’ beliefs on the system state. We show that these strategies are closed under the best-response mapping and that cooperation can be the optimal choice in some subsets of the state belief simplex. We characterize these equilibrium regions, prove uniqueness of a maximal equilibrium region, and devise an algorithm for its approximate computation. [ABSTRACT FROM AUTHOR]

This article presents ConVex optimization-based Stochastic steady-state Tracking Error Minimization (CV-STEM), a new state feedback control framework for a class of Itô stochastic nonlinear systems and Lagrangian systems. Its innovation lies in computing the control input by an optimal contraction metric, which greedily minimizes an upper bound of the steady-state mean squared tracking error of the system trajectories. Although the problem of minimizing the bound is nonconvex, its equivalent convex formulation is proposed utilizing SDC parameterizations of the nonlinear system equation. It is shown using stochastic incremental contraction analysis that the CV-STEM provides a sufficient guarantee for exponential boundedness of the error for all time with ${\bf \mathcal {L}_2}$ -robustness properties. For the sake of its sampling-based implementation, we present discrete-time stochastic contraction analysis with respect to a state- and time-dependent metric along with its explicit connection to continuous-time cases. We validate the superiority of the CV-STEM to PID, $\mathcal {H}_\infty$ , and baseline nonlinear controllers for spacecraft attitude control and synchronization problems. [ABSTRACT FROM AUTHOR]

Value functions derived from Markov decision processes arise as a central component of algorithms as well as performance metrics in many statistics and engineering applications of machine learning. Computation of the solution to the associated Bellman equations is challenging in most practical cases of interest. A popular class of approximation techniques, known as temporal difference (TD) learning algorithms, are an important subclass of general reinforcement learning methods. The algorithms introduced in this article are intended to resolve two well-known issues with TD-learning algorithms. Their slow convergence due to very high central limit theorem variance, and the fact that, for the problem of computing the relative value function, consistent algorithms exist only in special cases. First we show that the gradients of these value functions admit a representation that lends itself to algorithm design. Based on this result, a new class of differential TD-learning algorithms is introduced. For Markovian models on Euclidean space with smooth dynamics, the algorithms are shown to be consistent under general conditions. Numerical results show dramatic variance reduction in comparison to standard methods. [ABSTRACT FROM AUTHOR]

Probabilistic occupancy, the likelihood that the state at a known future time lies in a given set, is important in a variety of stochastic motion planning problems. We provide efficient computational techniques, based in Fourier transforms, to characterize the stochasticity of the future state for Markov jump affine systems. This class of systems captures a variety of important dynamics in planning problems, including the Dubins’ vehicle. We employ convex optimization to compute outer approximations of the superlevel sets of the probabilistic occupancy function, which is a key for preserving the safety guarantees sought in collision-avoidance problems. In contrast to traditional approaches, our approach does not rely on gridding, recursion, or sampling, accommodates non-Gaussian perturbed dynamics, and affords outer-approximation guarantees. We demonstrate our methods on the target pursuit problem with multiple robots pursuing a nonadversarial target with stochastic dynamics, and on the problem of computing keep-out regions for stochastic collision avoidance of a Dubins’ vehicle. [ABSTRACT FROM AUTHOR]

This paper presents a spectral approach to the uncertainty management in epidemic models through the formulation of chance-constrained stochastic optimal control problems. Specifically, a statistical moment-based polynomial expansion is used to calculate surrogate models of the stochastic state variables of the problem that allow for the efficient computation of their main statistics as well as their marginal and joint probability density functions at each time instant, which enable the uncertainty management in the epidemic model. Moreover, the surrogate models are employed to perform the corresponding sensitivity and risk analyses. The proposed methodology provides the designers of the optimal control policies with the capability to increase the predictability of the outcomes by adding suitable chance constraints to the epidemic model and formulating a proper cost functional. The chance-constrained optimal control of a COVID-19 epidemic model is considered in order to illustrate the practical application of the proposed methodology. • A spectral approach to the uncertainty management in epidemic models is proposed. • A statistical moment-based polynomial chaos expansion is employed. • The methodology is based on chance-constrained optimal control. • The approach also allows sensitivity and risk analyses to be performed. • The practical application is illustrated through a COVID-19 epidemic model. [ABSTRACT FROM AUTHOR]

This article considers the optimal control of the SIR model with both transmission and treatment uncertainty. It follows the model presented in Gatto and Schellhorn (2021). We make four significant improvements on the latter paper. First, we prove the existence of a solution to the model. Second, our interpretation of the control is more realistic: while in Gatto and Schellhorn (2021) the control α is the proportion of the population that takes a basic dose of treatment, so that α > 1 occurs only if some patients take more than a basic dose, in our paper, α is constrained between zero and one, and represents thus the proportion of the population undergoing treatment. Third, we provide a complete solution for the moderate infection regime (with constant treatment). Finally, we give a thorough interpretation of the control in the moderate infection regime, while Gatto and Schellhorn (2021) focused on the interpretation of the low infection regime. Finally, we compare the efficiency of our control to curb the COVID-19 epidemic to other types of control. • The optimal treatment policy is the same with or without rationing constraints, provided the maximum treatment quantity is not reached. • We provide full formulae for the optimal treatment policy in the moderate infection regime with constant treatment effectiveness. • The optimal policy is showed to yield better results in the COVID-19 pandemic. [ABSTRACT FROM AUTHOR]

• A stochastic optimal control is proposed to stabilize electrostatic MEMS resonators with noise disturbance. • The controller can be used to suppress the noise-activated pull-in instability. • The controller shows superior performance on rejecting the deterministic pull-in instability. • Time delay existing in the optimal control has a periodic impact on the control effectiveness. Noise-induced motions are a significant source of uncertainty in the response of electrostatic MEMS where electrical and mechanical sources contribute to noise and can result in sudden and dramatic loss of stability. In this manuscript, we present an analysis for the stabilization of an electrostatically actuated MEMS resonator in the presence of noise processes using a stochastic optimal control scheme. A stochastic lumped-mass model that accounts for the uncertainty in mass, mechanical restoring force, bias voltage, and AC voltage amplitude of the resonator is presented. The It o ^ equations, describing the averaged modulations of the resonator's total energy and phase difference, are obtained via the stochastic averaging of energy envelope. The stochastic optimal control law aiming at minimization the pull-in probability of the resonator is determined via the stochastic dynamic programming equations associated with the It o ^ equations. We show that the resulting stochastic optimal feedback control, with a careful selection of its control gain, can be used to suppress the noise-activated pull-in instability of the disturbed resonator. The impact of the time delay inherent in the feedback control input on the control effectiveness is investigated. In addition, the control scheme also shows superior performance in counteracting the deterministic pull-in instability of the resonator operating in the dynamic pull-in frequency band. Good agreement between the theoretical results and numerical simulation is demonstrated. [ABSTRACT FROM AUTHOR]