Search

Projection methods for computing stationary probability distributions for Markov chain models are presented. A general projection method is a method which seeks an approximation from a subspace of small dimension to the original problem. Thus, the original matrix problem of size N is approximated by one of dimension m, typically much smaller than N. A particularly successful class of methods based on this principle is that of Krylov subspace methods which utilize subspaces of the form span(v,av,...,A(exp m-1)v). These methods are effective in solving linear systems and eigenvalue problems (Lanczos, Arnoldi,...) as well as nonlinear equations. They can be combined with more traditional iterative methods such as successive overrelaxation, symmetric successive overrelaxation, or with incomplete factorization methods to enhance convergence.

In this paper, we present examples of a class of Markov chains that occur frequently, but whose associated matrices are a challenge to construct efficiently. These are Markov chains that arise as a result of several identical Markov chains running in parallel. Specifically for the cases considered, both the infinitesimal generator matrix for the continuous case, and more so the transition probability matrix for the discrete equivalent, are complex to construct effectively and efficiently. We summarize the algorithms for constructing the associated matrices and present examples of applications, ranging from special queueing problems to reliability issues and order statistics. MATLAB subroutines are provided in an online supplement for the implementation of the algorithms. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0759 .

This technical note contributes to the convergence analysis for iterative learning control (ILC) for linear stochastic systems under general data dropout environments, i.e., data dropouts occur randomly at both the measurement and actuator sides. Data updating in the memory array is arranged in such a way that data at every time instance is updated independently, which allows successive data dropouts both in time and iteration axes. The update mechanisms for both the computed input and real input are proposed and then the update process of both inputs is shown to be a Markov chain. By virtue of Markov modeling, a new analysis method is developed to prove the convergence in both mean square and almost sure senses. An illustrative example verifies the theoretical results.

The main focus of this paper is on the development of reliability measures for a repairable multi-state system which operates under dynamic regimes under the discrete-time hypothesis. The switching process of regimes is governed by a Markov chain, and the functioning process of the system follows another Markov chain with different transition probability matrices under different regimes. In terms of two chains as above, a new Markov chain is constructed to depict the evolution process of the dynamic system. For the regime consideration, some novel reliability indices are essential and firstly introduced in this paper. By means of hierarchical partitions for the new state space, Ion-Channel modeling theory and discrete-time Markov chain, the traditional and novel reliability and availability functions for the system under random regimes are easily obtained with the closed form solutions, such as two types of system reliabilities, two types of system point availabilities, two types of system multiple-point availabilities and the associated system multi-interval availabilities and so on. In addition, some probability distributions of sojourn times we are interested in are discussed and computed here. Finally, a numerical example is given to illustrate the results obtained in the paper.

In this paper, we prove that the Foster–Lyapunov drift condition is necessary and sufficient for recurrence of a Markov chain on a general state space. And then an answer in the affirmative for the question presented in the monograph of Meyn and Tweedie (Markov chains and stochastic stability. Springer, Berlin, 1993, P175) is obtained.

Motivated by various applications in queueing theory, this article is devoted to the stochastic monotonicity and comparability of Markov chains with block-monotone transition matrices. First, we introduce the notion of block-increasing convex order for probability vectors, and characterize the block-monotone matrices in the sense of the block-increasing order and block-increasing convex order. Second, we characterize the Markov chain with general transition matrix by martingale and provide a stochastic comparison of two block-monotone Markov chains under the two block-monotone orders. Third, the stochastic comparison results for the Markov chains corresponding to the discrete-time GI/G/1 queue with different service distributions under the two block-monotone orders are given, and the lower bound and upper bound of the Markov chain corresponding to the discrete-time GI/G/1 queue in the sense of the block-increasing convex order are found.

This paper deals with the H2 optimal control problem of discrete-time Markov jump linear systems (MJLS) considering the case in which the Markov chain takes values in a general Borel space S. It is assumed that the controller has access only to an output variable and to the jump parameter. The goal, in this case, is to design a dynamic Markov jump controller such that the H2-norm of the closed-loop system is minimised. It is shown that the H2-norm can be written as the sum of two H2-norms, such that one of them does not depend on the control, and the other one is obtained from the optimal filter for an infinite-horizon filtering problem. This result can be seen as a separation principle for MJLS with Markov chain in a Borel space S considering the infinite time horizon case.

In this note, the problem of event-based state estimation for a finite-state hidden Markov model under a generic stochastic event-triggering condition and an unreliable communication channel is investigated. The effect of packet dropout is characterized with a Gilbert–Elliott process. Utilizing the change of probability measure approach, the packet dropout model and the event-triggered measurement information available to the estimator, analytical expressions for the conditional probability distributions of the states are obtained, based on which the optimal event-based state estimates can be further calculated, together with a closed-form expression of the average sensor-to-estimator communication rate. The effectiveness of the proposed results is illustrated by an application to a wireless automated machine health monitoring problem.

The matrices of non-homogeneous Markov processes consist of time-dependent functions whose values at time form typical intensity matrices. For solvingsome problems they must be changed into stochastic matrices. A stochas-tic matrix for non-homogeneous Markov process consists of time-dependent functions, whose values are probabilities and it depend on assumed time pe- riod. In this paper formulas for these functions are derived. Although the formula is not simple, it allows proving some theorems for Markov stochastic processes, well known for homogeneous processes, but for non-homogeneous ones the proofs of them turned out shorter.

For a Markov chain described by an irreducible stochastic matrix A of order n, the mean first passage time m i , j measures the expected time for the Markov chain to reach state j for the first time given that the system begins in state i, thus quantifying the short-term behaviour of the chain. In this article, a lower bound for the maximum mean first passage time is found in terms of the stationary distribution vector of A, and some matrices for which equality is attained are produced. The main objective of this article is to characterise the directed graphs for which any stochastic matrix A respecting this directed graph attains equality in this lower bound, producing a class of Markov chains with optimal short-term behaviour.

In this paper, we consider an important practical industrial process identification problem where the time delay can change at every sampling instant. We model the time-varying discrete time-delay mechanism by a Markov chain model and estimate the Markov chain parameters along with the time-delay sequence simultaneously. Besides time-varying delay, processes with both time-invariant and time-variant model parameters are also considered. The former is solved by an expectation-maximization (EM) algorithm, while the latter is solved by a recursive version of the EM algorithm. The advantages of the proposed identification methods are demonstrated by numerical simulation examples and an evaluation on pilot-scale experiments.

The problem of averaging the kinetics of a two-stage absorbing Markov chain over random fluctuations in its forward transition probability approximated by the symmetric dichotomous stochastic process is solved exactly. It is shown that the temporal behavior of the population of chain’s transient state obeys a fourth-order differential equation with the tetra-exponential form of a solution given the finite frequency and mean amplitude of fluctuations. In the limit of frequent fluctuations, this tetra-exponential solution reduces to a simple bi-exponential form typical of the deterministic two-stage decay process lacking fluctuations in its transition probability. Rather, in the limit of rare fluctuations, the tetra-exponential solution, while simplifying to the tri- and bi-exponential solutions, becomes specific both for the low amplitude and the resonance amplitude fluctuations, respectively. Furthermore, there is a stochastic resonance point, where the forward transition probability is in resonance with the mean fluctuation amplitude, whereas the backward transition probability, decay transition probability, and fluctuation frequency are negligibly small. In result, the stochastic immobilization of the two-stage absorbing Markov chain in its initial state occurs at this point., Надано точного розв’язку проблемi усереднення кiнетики двостадiйного абсорбуючого ланцюга Маркова за випадковими флуктуацiями ймовiрностi прямого переходу, що апроксимується симетричним дихотомiчним стохастичним процесом. Показано, що часова поведiнка заселеностi перехiдного стану ланцюга задовольняє диференцiальне рiвняння четвертого порядку, чий розв’язок має чотириекспоненцiйний вигляд за скiнчених частоти та середньої амплiтуди флуктуацiй. У границi частих флуктуацiй цей чотириекспоненцiйний розв’язок зводиться до простого двоекспоненцiйного вигляду, що є типовим для детермiнiстичного процесу двостадiйного загасання, якому бракує флуктуацiй у ймовiрностi переходу. Проте у границi рiдких флуктуацiй чотириекспоненцiйний розв’язок, хоча й набуває спрощеного вигляду три- чи двоекспоненцiйного розв’язку, стає переважно специфiчним у вiдношеннi як до низько амплiтудних, так i до резонансно амплiтудних флуктуацiй. Бiльше того, iснує точка стохастичного резонансу, де ймовiрнiсть прямого переходу збiгається з середньою амплiтудою флуктуацiй, тодi як ймовiрностi загасання i зворотного переходу, а також частота флуктуацiй є незначними. Як результат, в цiй точцi здiйснюється стохастична iмобiлiзацiя двостадiйного абсорбуючого ланцюга Маркова у його початковому станi.

This paper is concerned with Schur-constant survival models for discrete random variables. Our main purpose is to prove that the associated partial sum process is a non-homogeneous Markov chain. This is shown in three different situations where the random variables considered take values in the sets 0, {0,1} or {0,…,m}, m ≥ 2. The property of Schur-constancy is also compared for these three cases.

A new mathematical model for discrete time series is proposed: homogenous vector Markov chain of the order s with partial connections. Conditional probability distribution for this model is determined only by a few components of previous vector states. Probabilistic properties of the model are given: ergodicity conditions and conditions under which the stationary probability distribution is uniform. Consistent statistical estimators for model parameters are constructed.

In this paper we contribute to the theory of conditional Markov chains (CMCs) that take finitely many values and that admit intensity. We provide a method for constructing a CMC with given intensity and with given conditional initial law, and which is also a doubly stochastic Markov chain. We provide a martingale characterization for such process, and we discuss other useful properties. We define and give sufficient and necessary conditions for strong Markovian consistency and weak Markovian consistency of a multivariate CMC. We use these results to model structured dependence between univariate CMCs, that is, to model a multivariate CMC whose components are univariate CMCs with given laws. An example of potential application of our theory is presented.

The issue of asynchronous passive control is addressed for Markov jump systems in this technical note. The asynchronization phenomenon appears between the system modes and controller modes, which is described by a hidden Markov model. Accordingly, a hidden Markov jump model is used to name the resultant closed-loop system. By utilizing the matrix inequality technique, three equivalent sufficient conditions are obtained, which can guarantee the hidden Markov jump systems to be stochastically passive. Based on the established conditions, the design of asynchronous controller, which covers the well-known mode-independent controller and synchronous controller as special cases, is addressed. The DC motor device is applied to demonstrate the practicability of the derived asynchronous synthesis scheme.

This paper focuses on a three-species food chain system which is formulated as stochastic differential equations with regime switching represented by a hidden Markov chain. Firstly, using the Wonham filter, we estimate the hidden Markov chain through the observable solution of the Markov chain in Gaussian white noise. Then two kinds of special dissipative control strategy are proposed to study the given model. That is, under $H_{\infty}$ control and passive control, the sufficient conditions for global asymptotic stability are established, respectively. Finally, numerical examples are given to illustrate the effectiveness of the theoretical results.

In this article, we consider some characterizations for the stationary distribution of Ornstein–Uhlenbeck process with a two-state Markov switching. We show that if the drift coefficients α(1), α(2...

Estimation of link delay densities in a computer network, from source-destination delay measurements, is of great importance in analyzing and improving the operation of the network. In this paper, we develop a general approach for estimating the density of the delay in any link of the network, based on continuous-time bivariate Markov chain modeling. The proposed approach also provides the estimates of the packet routing probability at each node, and the probability of each source-destination path in the network. In this approach, the states of one process of the bivariate Markov chain are associated with nodes of the network, while the other process serves as an underlying process that affects statistical properties of the node process. The node process is not Markov, and the sojourn time in each of its states is phase-type. Phase-type densities are dense in the set of densities with non-negative support. Hence, they can be used to approximate arbitrarily well any sojourn time distribution. Furthermore, the class of phase-type densities is closed under convolution and mixture operations. We adopt the expectation-maximization (EM) algorithm of Asmussen, Nerman, and Olsson for estimating the parameter of the bivariate Markov chain. We demonstrate the performance of the approach in a numerical study.

This paper considers continuous-time block-monotone Markov chains (BMMCs) and their block-augmented truncations. We first introduce the block monotonicity and block-wise dominance relation for continuous-time Markov chains, and then provide some fundamental results on the two notions. Using these results, we show that the stationary distribution vectors obtained by the block-augmented truncation converge to the stationary distribution vector of the original BMMC. We also show that the last-column-block-augmented truncation (LC-block-augmented truncation) provides the best (in a certain sense) approximation to the stationary distribution vector of a BMMC among all the block-augmented truncations. Furthermore, we present computable upper bounds for the total variation distance between the stationary distribution vectors of a Markov chain and its LC-block-augmented truncation, under the assumption that the original Markov chain itself may not be block-monotone but is block-wise dominated by a BMMC with exponential ergodicity. Finally, we apply the obtained bounds to a queue with a batch Markovian arrival process and state-dependent departure rates.

Markov state models have been widely applied to study time sequential events in a variety of disciplines. Due to their versatility for representing system stochasticity, Markov state models hold the promise to simplify simulation and design control policies for colloidal self-assembly systems. In this manuscript, we investigate the effects of state discretization, transition time, sampling approach, and the number of samples on the accuracy of a Markov state model for a colloidal self-assembly process. The model accuracy is evaluated based on the performance of the optimal control policy, calculated with a Markov decision process-based optimization framework, for controlling a Brownian dynamics simulation to produce perfect crystals. The results suggest using a dynamic sampling, a transition time similar to the system characteristic time, a clustering-based state discretization, and an average of at least five samples per state, to efficiently build an accurate Markov state model.

The increasing availability of opportunistic and dedicated sensors is transforming a once data-starved transport field into one of the most data-rich. While link-level travel time information can be derived or inferred from this data, methods for estimation of trip travel times between an origin and a destination pair are still evolving and limited, especially in the context of probability distribution estimation. This paper proposes a generalized Markov chain approach for estimating the probability distribution of trip travel times from link travel time distributions and takes into consideration correlations in time and space. The proposed approach consists of three major components, namely state definition, transition probabilities estimation and probability distribution estimation. A heuristic clustering method, based on Gaussian mixture models, has been developed to cluster link travel time observations with regard to their homogeneity and underlying traffic conditions. A transition probability estimation model is developed as a function of link characteristics and trip conditions using a logit model. By applying a Markov chain procedure, the probability distribution of trip travel times is estimated as the combination of Markov path travel time distributions weighted by their corresponding occurrence probabilities. The link travel time distribution is conditioned on the traffic conditions of the current link that can be estimated from historical observations. A moment generating function based algorithm is used to approximate the Markov path travel time distribution as the sum of correlated link travel time distributions conditional on traffic conditions. The proposed approach is applied in a transit case study using automatic vehicle location data. The results indicate that the method is effective and efficient, especially when correlations and multimodal distributions exist.

The markov prediction and analysis is a method based on markov chain theory, according to the present states and tendency of some variables, the possible states in a given period in the future will be forecasted. To determine the transition probability matrix is the crux of this method. This paper applies stochastic time series based on markov chain model to forecast the developing trends of China’s energy consumption structure under the influence of energy policy. This model is adopted to forecast the total amount of energy consumption and the trends of energy production and consumption structures. China’s energy consumption during 2006–2011 are simulated and its trends in 2015 and 2020 are forecast through this model. Results demonstrate that this model can effectively simulate and forecast the total amount and structures of primary energy consumption. Although the growth rate of energy consumption in China has decreased under the energy saving policy, but it not enough to achieve the objective of carbon reduction.

Deterministic epidemic models are attractive due to their compact nature, allowing substantial complexity with computational efficiency. This partly explains their dominance in epidemic modelling. However, the small numbers of infectious individuals at early and late stages of an epidemic, in combination with the stochastic nature of transmission and recovery events, are critically important to understanding disease dynamics. This motivates the use of a stochastic model, with continuous-time Markov chains being a popular choice. Unfortunately, even the simplest Markovian S-I-R model-the so-called general stochastic epidemic-has a state space of order [Formula: see text], where N is the number of individuals in the population, and hence computational limits are quickly reached. Here we introduce a hybrid Markov chain epidemic model, which maintains the stochastic and discrete dynamics of the Markov chain in regions of the state space where they are of most importance, and uses an approximate model-namely a deterministic or a diffusion model-in the remainder of the state space. We discuss the evaluation, efficiency and accuracy of this hybrid model when approximating the distribution of the duration of the epidemic and the distribution of the final size of the epidemic. We demonstrate that the computational complexity is [Formula: see text] and that under suitable conditions our approximations are highly accurate.

Focusing on stochastic dynamics involve continuous states as well as discrete events, this article investigates stochastic logistic model with regime switching modulated by a singular Markov chain involving a small parameter. This Markov chain undergoes weak and strong interactions, where the small parameter is used to reflect rapid rate of regime switching among each state class. Two-time-scale formulation is used to reduce the complexity. We obtain weak convergence of the underlying system so that the limit has much simpler structure. Then we utilize the structure of limit system as a bridge, to invest stochastic permanence of original system driving by a singular Markov chain with a large number of states. Sufficient conditions for stochastic permanence are obtained. A couple of examples and numerical simulations are given to illustrate our results.

Publisher Summary This chapter focuses on the continuous-time analog probability models of the Markov chains that have a wide variety of applications in the real world. The Poisson process is a continuous-time Markov chain having states 0, 1, 2, … and always proceeds from state n to state n + 1, where n ≥ 0. A continuous-time Markov chain is a stochastic process that moves from state to state in accordance with a discrete-time Markov chain, but is such that the amount of time it spends in each state, before proceeding to the next state, is exponentially distributed. In addition, the amount of time the process spends in state i and the next state visited must be independent random variables.

Publisher Summary This chapter focuses on Markov chains of the probability theory. It presents a stochastic process {Xn, n = 0, 1, 2…} that takes on a finite or countable number of possible values. This set of possible values of the process is denoted by the set of nonnegative integers {0, 1, 2…}. If Xn = i, the process is said to be in state i at time n. Such a stochastic process is known as a Markov chain. The value Pij represents the probability that the process will, when in state i, next make a transition into state j. The Chapman–Kolmogorov equations provide a method for computing these n-step transition probabilities. In a finite-state Markov chain, all recurrent states are positive recurrent.

In previous work, the embedding problem is examined within the entire set of discrete-time Markov chains. However, for several phenomena, the states of a Markov model are ordered categories and the transition matrix is state-wise monotone. The present paper investigates the embedding problem for the specific subset of state-wise monotone Markov chains. We prove necessary conditions on the transition matrix of a discrete-time Markov chain with ordered states to be embeddable in a state-wise monotone Markov chain regarding time-intervals with length 0.5: A transition matrix with a square root within the set of state-wise monotone matrices has a trace at least equal to 1.

It was conjectured in the paper “Stationary probability vectors of higher-order Markov chains” (Li and Zhang, 2015 [7] ) that if the set of stationary vectors of the second-order Markov chain contains k-interior points of the ( k − 1 ) -dimensional face of the simplex Ω n then every vector in the ( k − 1 ) -dimensional face is a stationary vector. In this paper, we provide counterexamples to this conjecture.

In this paper, we consider the long time behavior of Cox–Ingersoll–Ross (CIR) interest rate model with Markov switching. Using the ergodic theory of switching diffusions, we show that CIR model with Markov switching has a unique stationary distribution. Furthermore, we prove that the sequence X ¯ t : = 1 t ∫ 0 t X s d s converges almost surely. As a by-product, we find that the marginal stationary distribution for CIR model with Markov switching can be determined uniquely by its moments.

A novel model referred to as two-dimensional continuous 3 × 3 order hidden Markov model is put forward to avoid the disadvantages of the classical hypothesis of two-dimensional continuous hidden Markov model. This paper presents three equivalent definitions of the model, in which the state transition probability relies on not only immediate horizontal and vertical states but also immediate diagonal state, and in which the probability density of the observation relies on not only current state but also immediate horizontal and vertical states. The paper focuses on the three basic problems of the model, namely probability density calculation, parameters estimation and path backtracking. Some algorithms solving the questions are theoretically derived, by exploiting the idea that the sequences of states on rows or columns of the model can be viewed as states of a one-dimensional continuous 1 × 2 order hidden Markov model. Simulation results further demonstrate the performance of the algorithms. Because there are more statistical characteristics in the structure of the proposed new model, it can more accurately describe some practical problems, as compared to two-dimensional continuous hidden Markov model.

The Markov property of driving cycles was discussed by making a thorough description of their essential characteristics. The Markov chain, a useful tool for designing and expressing driving cycles, has been increasingly used in the field of driving cycles in recent years. Although the Markov property of driving cycles left unproved, some researchers have taken it for granted that it is appropriate to design driving cycle with Markov chain. In our research, the vehicle dynamics model and the car-following model were used to establish the two-dimensional Markov state transition model. On the other hand, the driving data from the city of Changchun were collected to analyze how states (in Markov theory) correlation changes with the increase of time intervals. After the Markov property had been proven, the theory of ergodicity was applied to reveal the relationship between velocity-acceleration joint probability distribution (VA Probability) and state transition matrix. Finally, the application of Markov property was also discussed briefly. This research will lay a theoretical foundation for designing driving cycles and ECO driving (Economical and Ecological).

A novel model is proposed to overcome the shortages of the classical hypothesis of the two-dimensional discrete hidden Markov model. In the proposed model, the state transition probability depends on not only immediate horizontal and vertical states but also on immediate diagonal state, and the observation symbol probability depends on not only current state but also on immediate horizontal, vertical and diagonal states. This paper defines the structure of the model, and studies the three basic problems of the model, including probability calculation, path backtracking and parameters estimation. By exploiting the idea that the sequences of states on rows or columns of the model can be seen as states of a one-dimensional discrete 1 × 2 order hidden Markov model, several algorithms solving the three questions are theoretically derived. Simulation results further demonstrate the performance of the algorithms. Compared with the two-dimensional discrete hidden Markov model, there are more statistical characteristics in the structure of the proposed model, therefore the proposed model theoretically can more accurately describe some practical problems.

In 2013, Lu and Ren \cite {luren} considered anticipated backward stochastic differential equations driven by finite state, continuous time Markov chain noise and established the existence and uniqueness of the solutions of these equations and a scalar comparison theorem. In this paper, we provide an estimate for their solutions and study the duality between these equations and stochastic differential delayed equations with Markov chain noise. Finally we derive another comparison theorem for these solutions depending only on the two drivers., 24 pages

Markov models are ubiquitously used to represent the function of single ion channels. However, solving the inverse problem to construct a Markov model of single channel dynamics from bilayer or patch-clamp recordings remains challenging, particularly for channels involving complex gating processes. Methods for solving the inverse problem are generally based on data from voltage clamp measurements. Here, we describe an alternative approach to this problem based on measurements of voltage traces. The voltage traces define probability density functions of the functional states of an ion channel. These probability density functions can also be computed by solving a deterministic system of partial differential equations. The inversion is based on tuning the rates of the Markov models used in the deterministic system of partial differential equations such that the solution mimics the properties of the probability density function gathered from (pseudo) experimental data as well as possible. The optimization is done by defining a cost function to measure the difference between the deterministic solution and the solution based on experimental data. By evoking the properties of this function, it is possible to infer whether the rates of the Markov model are identifiable by our method. We present applications to Markov model well-known from the literature.

Consider a finite-state, time-continuous, homogeneous Markov chain with an absorbing state. The time to absorption in such a state is said to have a phase-type distribution. Markov models are useful because they may incorporate an understanding of how a phenomenon, for example, a disease, moves through different stages. However, for tractability and to maintain the Markov property, the stages are modeled with exponential waiting times. Aalen[1] introduced such phase-type distributions based on Markov processes for modeling disease progression in survival analysis. Huzurbazar[2] extended this work by generalizing these models to semi-Markov processes with nonexponential waiting times, called generalized phase-type distributions. The generalization allows more realistic modeling of the stages of the disease where the Markov property and exponential waiting times may not hold. Flowgraph models are used to provide a closed-form solution for the distributions in situations where involving nonexponential waiting times. Flowgraph models are a tool that aid in the analysis of semi-Markov processes. Densities and hazard rates of phase-type distributions can be calculated from standard Markov chain theory. Keywords: survival analysis; flowgraph model; hazard rate; Markov chain; quasi-stationarity; multistate model

The Lyapunov---Krasovskii second method is used to obtain the sufficient conditions for asymptotic stochastic global stability, global stability, mean square stability of trivial solutions of systems of stochastic diffusion functional-differential equations with Markov switching, and the theory is illustrated using two model problems.

Bootstrapping time series is one of the most acknowledged tools to study the statistical properties of an evolutive phenomenon. An important class of bootstrapping methods is based on the assumption that the sampled phenomenon evolves according to a Markov chain. This assumption does not apply when the process takes values in a continuous set, as it frequently happens with time series related to economic and financial phenomena. In this paper we apply the Markov chain theory for bootstrapping continuous-valued processes, starting from a suitable discretization of the support that provides the state space of a Markov chain of order \(k \ge 1\). Even for small k, the number of rows of the transition probability matrix is generally too large and, in many practical cases, it may incorporate much more information than it is really required to replicate the phenomenon satisfactorily. The paper aims to study the problem of compressing the transition probability matrix while preserving the “law” characterising the process that generates the observed time series, in order to obtain bootstrapped series that maintain the typical features of the observed time series. For this purpose, we formulate a partitioning problem of the set of rows of such a matrix and propose a mixed integer linear program specifically tailored for this particular problem. We also provide an empirical analysis by applying our model to the time series of Spanish and German electricity prices, and we show that, in these medium size real-life instances, bootstrapped time series reproduce the typical features of the ones under observation.

The length of the stochastic shortest path is defined as the arrival probability from a source node to a destination node. The uncertainty of the network topology causes unstable connections between nodes. A discrete-time Markov chain is devised according to the uniform distribution of existing arcs where the arrival probability is computed as a finite transition probability from the initial state to the absorbing state. Two situations are assumed, departing from the current state to a new state, or waiting in the current state while expecting better conditions. Our goal is to contribute to determining the critical node in a stochastic network, where its absence results in the greatest decrease of the arrival probability. The proposed method is a simply application for analyzing the resistance of networks against congestion and provides some crucial information of the individual nodes. Finally, this is illustrated using networks of various topologies.

A one-dimensional diffusion process is considered. This process is proposed to have the homogeneous Markov property with respect to the first exit time from any open interval (semi-Markov property). A diffusion property of the process is defined as asymptotical equiprobable exit through each of two edges of any symmetric neighborhood interval of an initial point of a sample process trajectory while length of the neighborhood tends to zero. Such a process is proved to have a limit as $t\to\infty$ if probability of the process not leaving this neighborhood is decreased as square of its length. In particular this condition is satisfied for a diffusion Markov process with a break for which the nonexit condition is replaced by the break condition. A semi-Markov method is applied to a derivation of the formula of a conditional final distribution of the diffusion process with a limit at infinity.

A Glivenko–Cantelli theorem is a fundamental result in statistics. It says that an empirical distribution function uniformly approximates the true distribution function for a sufficiently large sample size. We prove general Glivenko–Cantelli theorems for three types of sequences of random variables: delayed regenerative, delayed stationary and delayed exchangeable. In particular, our results hold for irreducible Harris recurrent Markov chains that admit a stationary probability distribution but are not necessarily in the stationary state. We also do not assume any mixing conditions on the Markov chain. This is useful in the application of Markov chain Monte Carlo methods. A key tool used is a generalized version of Polya's theorem on the convergence of distribution functions. Copyright © 2016 John Wiley & Sons, Ltd.

We consider the problem of risk-sensitive stochastic control under a Markov modulated denial-of-service (DoS) attack strategy in which the attacker, using a hidden Markov model, stochastically jams the control packets in the system. For a discrete-time partially observed stochastic system with an exponential running cost, we provide a solution in terms of the finite-dimensional dynamics of the system through a chain of measure transformation techniques. We also prove a separation principle under which a recursive optimal control policy together with a newly defined information-state constitutes an equivalent completely observable stochastic control problem. Remarkably, on the transformed measure space, the solution to the optimal control problem appears as if it depends only on the sample-path (or path-estimation) of the DoS attack sequences in the system.

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

The article is devoted to the estimation of the convergence rate of integral functionals of a Markov process. Under the assumption that the given Markov process admits a transition probability density differentiable in $t$ and the derivative has an integrable upper bound of a certain type, we derive the accuracy rates for strong and weak approximations of the functionals by Riemannian sums. We also develop a version of the parametrix method, which provides the required upper bound for the derivative of the transition probability density for a solution of an~SDE driven by a locally stable process. As an application, we give accuracy bounds for an~approximation of the price of an occupation time option.

This chapter proposes an auxiliary particle filter algorithm for inference in regime switching stochastic volatility models in which the regime state is governed by a first-order Markov chain. It proposes an ongoing updated Dirichlet distribution to estimate the transition probabilities of the Markov chain in the auxiliary particle filter. A simulation-based algorithm is presented for the method that demonstrates the ability to estimate a class of models in which the probability that the system state transits from one regime to a different regime is relatively high. The methodology is implemented in order to analyze a real-time series, namely, the foreign exchange rate between the Australian dollar and the South Korean won.

Regime-switching models, in particular Hidden Markov Models (HMMs) where the switching is driven by an unobservable Markov chain, are widely-used in financial applications, due to their tractability and good econometric properties. In this work we consider HMMs in continuous time with both constant and switching volatility. In the continuous-time model with switching volatility the underlying Markov chain could be observed due to this stochastic volatility, and no estimation (filtering) of it is needed (in theory), while in the discretized model or the model with constant volatility one has to filter for the underlying Markov chain. The motivations for continuous-time models are explicit computations in finance. To have a realistic model with unobservable Markov chain in continuous time and good econometric properties we introduce a regime-switching model where the volatility depends on the filter for the underlying chain and state the filtering equations. We prove an approximation result for a fixed information filtration and further motivate the model by considering social learning arguments. We analyze its relation to the switching volatility model and present a convergence result for the discretized model. We then illustrate its econometric properties by considering numerical simulations.