Search

We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with the history process serving as the controlled state process and a further refinement in which the control actions and the state process are causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies under the discounted cost and average cost infinite horizon criteria. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. For the average cost setup, we provide new existence conditions and also a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for nonlinear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity. [ABSTRACT FROM AUTHOR]

We propose a general framework for the study of the genealogy of neutral discrete-time populations. We remove the standard assumption of exchangeability of offspring distributions appearing in Cannings models, and replace it by a less restrictive condition of non-heritability of reproductive success. We provide a general criterion for the weak convergence of their genealogies to Ξ -coalescents, and apply it to a simple parametrization of our scenario (which, under mild conditions, we also prove to essentially include the general case). We provide examples for such populations, including models with highly-asymmetric offspring distributions and populations undergoing random but recurrent bottlenecks. Finally we study the limit genealogy of a new exponential model which, as previously shown for related models and in spite of its built-in (fitness) inheritance mechanism, can be brought into our setting. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a fractional analogue of the Wiener polynomial chaos expansion. It is important to highlight that the fractional order relates to the order of chaos decomposition elements, and not to the process itself, which remains the standard Wiener process. The central instrument in our fractional analogue of the Wiener chaos expansion is the function denoted as H α (x , y) , referred to herein as a power-normalised parabolic cylinder function. Through careful analysis of several fundamental deterministic and stochastic properties, we affirm that this function essentially serves as a fractional extension of the Hermite polynomial. In particular, the power-normalised parabolic cylinder function with the Wiener process and time as its arguments, H α (W t , t) , demonstrates martingale properties and can be interpreted as a fractional Itô integral with 1 as the integrand, thereby drawing parallels with its non-fractional counterpart. To build a fractional analogue of polynomial Wiener chaos on the real line, we introduce a new function, which we call the extended Hermite function, by smoothly joining two power-normalized parabolic cylinder functions at zero. We form an orthogonal set of extended Hermite functions as a one-parameter family and use tensor products of the extended Hermite functions as building blocks in the fractional Wiener chaos expansion, in the same way that tensor products of Hermite polynomials are used as building blocks in the Wiener chaos polynomial expansion. [ABSTRACT FROM AUTHOR]

This paper extends the results provided in Jasso-Fuentes et al. (Appl Math Optim 81(2):409–441, 2020b) and Jasso-Fuentes et al. (Pure Appl Funct Anal 9(3):675–704, 2024) regarding the study of discrete-time hybrid stochastic models with general spaces and total discounted payoffs. This extension incorporates the handling of negative and/or unbounded costs per stage. In particular, it encompasses interesting applications, such as scenarios where the controller optimizes net costs, social welfare costs, or distances between points. These situations arise when assumptions of both non-negativeness and boundedness on the cost per stage do not apply. Our proposal relies on Lyapunov-like conditions, enabling, among other aspects, the finiteness of the value function and the existence of solutions to the associated dynamic programming equation. This equation is crucial for deriving optimal control policies. To illustrate our theory, we include an example in inventory-manufacturing management, highlighting its evident hybrid nature. [ABSTRACT FROM AUTHOR]

In this paper, a stochastic Lotka-Volterra system with infinite delay is considered. A new concept of extinction, namely, the almost sure β-extinction is proposed and sufficient conditions for the solution to be almost sure β-extinction are obtained. When the positive equilibrium exists and the intensities of the noises are small enough, any solution of the system is attracted by the positive equilibrium. Finally, numerical simulations are carried out to support the results. [ABSTRACT FROM AUTHOR]

In this paper, random environment and random time delay are introduced into general nonlinear autoregressive models to improve the applicability of these models. The introduction of random environment makes nonlinear autoregressive models more flexible and more widely used. Random time delay is a new way for models to be applied in new fields. In order to analyze the properties of the proposed new model, we construct a new stochastic process and prove that it is an irreducible and aperiodic time-homogeneous Markov chain. The nonrecurrence of the proposed new type of models is considered, and some sufficient conditions for nonrecurrence are investigated. [ABSTRACT FROM AUTHOR]

In this work, we characterize the fixed points of the doubly stochastic operators defined on discrete spaces $ \ell ^p(I) , $ ℓ p (I) , where I is a nonempty set and $ p\in [1,+\infty). $ p ∈ [ 1 , + ∞). We use our results to characterize all Markov chains with infinite state space and doubly stochastic transition probability matrix that have at least one stationary distribution vector. [ABSTRACT FROM AUTHOR]

We give rates of convergence in the almost sure invariance principle for sums of dependent random variables with semi-exponential tails, whose coupling coefficients decrease at a sub-exponential rate. We show that the rates in the strong invariance principle are in powers of log n . We apply our results to iid products of random matrices. [ABSTRACT FROM AUTHOR]

This paper investigates the asymptotic analysis of the hitting time of Markov-type jump processes (i.e., semi-Markov, Markov, in continuous or discrete time) with a small probability of entering a non-empty terminal subset. This means that absorption is a rare event. The mean hitting time function of all four type processes obeyed the same equation. We obtain unified results of asymptotic approximation in a series scheme or, equivalently, a functional type of mean hitting time. [ABSTRACT FROM AUTHOR]

This paper studies constrained Markov decision processes under the total expected discounted cost optimality criterion, with a state-action dependent discount factor that may take any value between zero and one. Both the state and the action space are assumed to be Borel spaces. By using the linear programming approach, consisting in stating the control problem as a linear problem on a set of occupation measures, we show the existence of an optimal stationary Markov policy. Our results are based on the study of both weak-strong topologies in the space of occupation measures and Young measures in the space of Markov policies. [ABSTRACT FROM AUTHOR]

In this paper, we consider a subclass of piecewise deterministic Markov processes with a Polish state space that involve a deterministic motion punctuated by random jumps, occurring in a Poisson-like fashion with some state-dependent rate, between which the trajectory is driven by one of the given semiflows. We prove that there is a one-to-one correspondence between stationary distributions of such processes and those of the Markov chains given by their post-jump locations. Using this result, we further establish a criterion guaranteeing the existence and uniqueness of the stationary distribution in a particular case, where the post-jump locations result from the action of a random iterated function system with an arbitrary set of transformations. [ABSTRACT FROM AUTHOR]

This article studies deep neural network expression rates for optimal stopping problems of discrete-time Markov processes on high-dimensional state spaces. A general framework is established in which the value function and continuation value of an optimal stopping problem can be approximated with error at most ε by a deep ReLU neural network of size at most κ d q ε − r . The constants κ , q , r ≥ 0 do not depend on the dimension d of the state space or the approximation accuracy ε . This proves that deep neural networks do not suffer from the curse of dimensionality when employed to approximate solutions of optimal stopping problems. The framework covers for example exponential Lévy models, discrete diffusion processes and their running minima and maxima. These results mathematically justify the use of deep neural networks for numerically solving optimal stopping problems and pricing American options in high dimensions. [ABSTRACT FROM AUTHOR]

In this paper, we prove the following assertion for an absorbing Markov decision process (MDP) with the given initial distribution, which is also assumed to be semi-continuous: the continuity of the projection mapping from the space of strategic measures to the space of occupation measures, both endowed with their weak topologies, is equivalent to the MDP model being uniformly absorbing. An example demonstrates, among other interesting scenarios, that for an absorbing (but not uniformly absorbing) semi-continuous MDP with the given initial distribution, the space of occupation measures can fail to be compact in the weak topology. [ABSTRACT FROM AUTHOR]

A Hidden Markov Model (HMM) is a couple of stochastic processes: A hidden Markov process and an observed emission process. Generally, the HMMs are used to study the hidden behavior of random systems through some observed emission sequences generated by the phenomenon under study. In this frame-work, we propose to solve the likelihood and the decoding problems of HMMs whose state space is composed of a continuous part and a discrete part. We adapt forward, backward and Viterbi algorithms to the case of our proposal. Numerical examples and Monte Carlo simulations are considered to show the efficiency and the adaptation of the algorithms for the proposed model. [ABSTRACT FROM AUTHOR]

In this paper, we consider (upward skip-free) discrete-time and discrete-space Markov additive chains (MACs) and develop the theory for the so-called W ~ and Z ~ scale matrices, which are shown to play a vital role in the determination of a number of exit problems and related fluctuation identities. The theory developed in this fully discrete set-up follows similar lines of reasoning as the analogous theory for Markov additive processes in continuous time and is exploited to obtain the probabilistic construction of the scale matrices, identify the form of the generating function and produce a simple recursion relation for W ~ , as well as its connection with the so-called occupation mass formula. In addition to the standard one- and two-sided exit problems (upwards and downwards), we also derive distributional characteristics for a number of quantities related to the one- and two-sided 'reflected' processes. [ABSTRACT FROM AUTHOR]

The transition kernel of an ℝn-valued diffusion or jump diffusion process {Xt} is known to satisfy the Feller property if {Xt} is the solution of an SDE whose coefficients are Lipschitz continuous. This Lipschitz route to Feller falls short if {Xt} is the solution of an SDE whose coefficients depend on a state-dependent regime-switching process {θt}. In this paper it is shown that pathwise uniqueness and the Feller property are satisfied under mild conditions for a regime-switching jump diffusion process {Xt, θt} with hybrid jumps, i.e. jumps in {Xt} that occur simultaneously with {θt} switching. [ABSTRACT FROM AUTHOR]

In this paper, we give sufficient conditions for the almost sure central limit theorem started at a point, known under the name of quenched central limit theorem. This is achieved by using a new idea of conditioning with respect to both the past and the future of the Markov chain. As applications, we provide a new sufficient projective condition for the quenched CLT. [ABSTRACT FROM AUTHOR]

Let (gn)n⩾l be a sequence of independent and identically distributed positive random d × d matrices and consider the matrix product Gn = gn⋯ g1. Under suitable conditions, we establish the Berry-Esseen bounds on the rate of convergence in the central limit theorem and Cramér-type moderate deviation expansions, for any matrix norm ∥Gn∥ of Gn, its entries G n i , j and its spectral radius ρ(Gn). Extended versions of their joint law with the direction of the random walk Gnx are also established, where x is a starting point in the unit sphere of ℝd. [ABSTRACT FROM AUTHOR]

We investigate the asymptotic behaviour of the sequence of forward type iterations of a given random-valued vector function on the state space being a separable and complete metric space. Assuming non-linear contraction in mean we prove that the considered sequence converges weakly to a random variable with a finite first moment and independent of the initial state. Moreover, we show that the speed of this convergence does not have to be geometric. We also present examples illustrating the result obtained. [ABSTRACT FROM AUTHOR]

Motivated by queueing applications, we study various reflected autoregressive processes with dependencies. Among others, we study cases where the interarrival and service times are proportionally dependent on additive and/or subtracting delay, as well as cases where interarrival times depend on whether the service duration of the previous arrival exceeds or not a random threshold. Moreover, we study cases where the autoregressive parameter is constant as well as a discrete or a continuous random variable. More general dependence structures are also discussed. Our primary aim is to investigate a broad class of recursions of autoregressive type for which several independence assumptions are lifted and for which a detailed exact analysis is provided. We provide expressions for the Laplace transform of the waiting time distribution of a customer in the system in terms of an infinite sum of products of known Laplace transforms. An integer-valued reflected autoregressive process that can be used to model a novel retrial queueing system with impatient customers and a general dependence structure is also considered. For such a model, we provide expressions for the probability generating function of the stationary orbit queue length distribution in terms of an infinite sum of products of known generating functions. A first attempt towards a multidimensional setting is also considered. [ABSTRACT FROM AUTHOR]

Polar slice sampling, a Markov chain construction for approximate sampling, performs, under suitable assumptions on the target and initial distribution, provably independent of the state space dimension. We extend the aforementioned result of Roberts and Rosenthal (Stoch Model 18(2):257–280, 2002) by developing a theory which identifies conditions, in terms of a generalized level set function, that imply an explicit lower bound on the spectral gap even in a general slice sampling context. Verifying the identified conditions for polar slice sampling yields a lower bound of 1/2 on the spectral gap for arbitrary dimension if the target density is rotationally invariant, log-concave along rays emanating from the origin and sufficiently smooth. The general theoretical result is potentially applicable beyond the polar slice sampling framework. [ABSTRACT FROM AUTHOR]

In this study, we consider disorder detection problems for statistical models with dependent observations defined by Markov chains in a Bayesian setting for a uniform prior distribution when the number of observations is limited by some known quantity. To achieve this, a new optimal sequential detection procedure is constructed using optimal stopping methods. This procedure minimizes the average delay time in the class of sequential procedures with false alarm probabilities that do not exceed a fixed value. The main difference between the proposed detection procedure and the conventional ones is that it is based not on the posterior probabilities but on the weighted Shiryaev-Roberts statistic. This allows for optimal detection in a nonasymptotic sense over any bounded time interval. Thereafter, we applied the constructed procedures to the early detection problem for the beginning of the epidemic spread. We used two epidemic models: the binomial model proposed by Baron, Choudhary, and Yu (2013) and the model based on the Gaussian approximation introduced by Pergamenchtchikov, Tartakovsky, and Spivak (2022). The theoretical results were confirmed through numerical simulations using the Monte Carlo method. [ABSTRACT FROM AUTHOR]

We study a natural random walk on the n × n uni-upper triangular matrices, with entries in Z / m Z , generated by steps which add or subtract a uniformly random row to the row above. We show that the mixing time of this random walk is O (m 2 n log n + n 2 m o (1)) . This answers a question of Stong and of Arias-Castro, Diaconis, and Stanley. [ABSTRACT FROM AUTHOR]

We study the ergodic behaviour of a discrete-time process X which is a Markov chain in a stationary random environment. The laws of X t are shown to converge to a limiting law in (weighted) total variation distance as t → ∞ . Convergence speed is estimated, and an ergodic theorem is established for functionals of X. Our hypotheses on X combine the standard "drift" and "small set" conditions for geometrically ergodic Markov chains with conditions on the growth rate of a certain "maximal process" of the random environment. We are able to cover a wide range of models that have heretofore been intractable. In particular, our results are pertinent to difference equations modulated by a stationary (Gaussian) process. Such equations arise in applications such as discretized stochastic volatility models of mathematical finance. [ABSTRACT FROM AUTHOR]

This paper is devoted to the analysis of the finite-dimensional distributions and asymptotic behavior of extremal Markov processes connected with the Kendall convolution. In particular, we provide general formulas for the finite dimensional distributions of the random walk driven by the Kendall convolution for a large class of step size distributions. Moreover, we prove limit theorems for random walks and associated continuous-time stochastic processes. [ABSTRACT FROM AUTHOR]