Search

We propose an online algorithm for solving a class of continuous-state Markov decision processes. The algorithm combines classical Q-learning with an asynchronous averaging procedure, which allows Q-function estimates at sampled state–action pairs to be adaptively updated based on observations collected along a single sample trajectory. These estimates are then used to iteratively construct an interpolation-based function approximator of the Q-function. We prove the convergence of the algorithm and provide numerical results to illustrate its performance. • Proposed a model-free Q-learning algorithm for solving a class of infinite horizon Markov decision processes with continuous state spaces. • Proved the strong convergence of the algorithm when used in conjunction with a class of non-linear function approximators. • Illustrated the performance of the proposed algorithm through simulation comparison studies. [ABSTRACT FROM AUTHOR]

This paper is concerned with a class of coupled forward–backward stochastic differential equations (FBSDEs, for short) involving time delays and time advancements on infinite horizon. By introducing a randomized Lipschitz condition and a randomized monotonicity condition, the unique solvability of FBSDEs is obtained. Then the theoretical result is applied to a linear–quadratic (LQ, for short) problem of a time-delayed system with random coefficients. An explicit expression of the unique optimal control is obtained. [ABSTRACT FROM AUTHOR]

Abstract: This paper is concerned with one kind of forward–backward linear quadratic stochastic control problem whose system is described by a linear anticipated forward–backward stochastic differential delayed equation. The explicit form of the optimal control is derived. Optimal state feedback regulators are studied in two special cases. For the case with delay in just the control variable, the optimal state feedback regulator is obtained by the Riccati equation. For the other case with delay in just the state variable, the optimal state feedback regulator is analyzed by the value function approach. [Copyright &y& Elsevier]

For an optimal control problem of an Itô's type stochastic differential equation, the control process could be taken in open-loop or closed-loop forms. In the standard literature, provided appropriate regularity, the value functions under these two types of controls are equal and are the unique (viscosity) solution to the corresponding (path-dependent) HJB equation. In this short note, we provide a counterexample in the path dependent setting showing that these value functions can be different in general. [ABSTRACT FROM AUTHOR]

In this letter we provide several informative tight error bounds when using value function approximators for the risk-sensitive cost setting for a given policy represented using exponential utility. The novelty of our approach is that we make use of the irreducibility of the underlying Markov chain (resulting in better bounds using Perron–Frobenius eigenvectors) to derive new bounds whereas the earlier work used primarily the spectral variation bound which holds for any matrix, hence did not make use of the irreducibility. All our bounds have a perturbation term for large state spaces. We also present examples where we show that the new bounds perform 90-100% better than the earlier proposed spectral variation bound. [ABSTRACT FROM AUTHOR]

In this paper, an open problem is solved, for the stochastic optimal control problem with delay where the control domain is nonconvex and the diffusion term contains both control and its delayed term. Inspired by previous results about delayed stochastic control systems, Peng's global stochastic maximum principle is generalized to the time delayed case. A special backward stochastic differential equation is introduced to deal with the cross terms, when applying the duality technique. Comparing with the classical result, the maximum condition contains an indicator function, which in fact is the characteristic of the stochastic optimal control problem with delay. Furthermore, to illustrate the applications of our theoretical results, three dynamic optimization problems are addressed based on the global maximum principle. [ABSTRACT FROM AUTHOR]