Search

This paper presents a compositional framework for the modeling of interactive continuous-time Markov chains with time-dependent rates, a subclass of communicating piecewise deterministic Markov processes. A poly-time algorithm is presented for computing the coarsest quotient under strong bisimulation for rate functions that are either piecewise uniform or (piecewise) polynomial. Strong as well as weak bisimulation are shown to be congruence relations for the compositional framework, thus allowing component-wise minimization. In addition, a new characterization of transient probabilities in time-inhomogeneous Markov chains with piecewise uniform rates is provided.

In this paper we consider discrete-time multidimensional Markov chains having a block transition probability matrix which is the sum of a matrix with repeating block rows and a matrix of upper-Hessenberg, quasi-Toeplitz structure. We derive sufficient conditions for the existence of the stationary distribution, and outline two algorithms for calculating the stationary distribution.

A Markov chain model is constructed to simulate pattern formation arising from an evolutionary population of interactive homogeneous agents. In the structure of optimal evolution and stochastic properties, the model exhibits emergent properties with rich dynamical diagrams. We study the cluster splitting transition with the coarsening period-adding phenomena shown by the model, which gives an example of pattern formation in the evolution of complex system and reveals dynamical behavior of the Markov chain process.

We arc concerned here with properties of the Markov chains associated to some finite order linear recursive sequences of nonnegative coefficients. The stochastic matrix formulation of these sequences permits to go thoroughly into their connection with the discrete parameter Markov chains theory, which give arise to some results of convergence. Moreover, we compute the generating function of the random variable defining the time of absorption of a stochastic process, of finite set of transient states, by a class of recurrent states.

We consider the M/G/1 and GI/M/1 types of Markov chains for which their one step transitions depend on the times of the transitions. These types of Markov chains are encountered in several stochastic models, including queueing systems, dams, inventory systems, insurance risk models, etc. We show that for the cases when the time parameters are periodic the systems can be analyzed using some extensions of known results in the matrix-analytic methods literature. We have limited our examples to those relating to queueing systems to allow us a focus. An example application of the model to a real life problem is presented.

Information theoretic methods are used to prove convergence in information divergence of reversible Markov chains. Also some ergodic theorems for information divergence are proved.

A matrix analytic paradigm, termed Quasi-Birth-Death Markov chains on binomial-like trees, is introduced and a quadratically converging algorithm to assess its steady state is presented. In a bivariate Markov chain {(X t ,N t ),t¡Ý0}, the values of the variable X t are nodes of a binomial-like tree of order d, where the ith child has i children of its own. We demonstrate that it suffices to solve d quadratic matrix equations to yield the steady state vector, the form of which is matrix geometric. We apply this framework to analyze the multilevel feedback scheduling discipline, which forms an essential part in contemporary operating systems.

In this work, a method is presented for analysis of Markov chains modeling evolutionary algorithms through use of a suitable quotient construction. Such a notion of quotient of a Markov chain is frequently referred to as "coarse graining" in the evolutionary computation literature. We shall discuss the construction of a quotient of an irreducible Markov chain with respect to an arbitrary equivalence relation on the state space. The stationary distribution of the quotient chain is "coherent" with the stationary distribution of the original chain. Although the transition probabilities of the quotient chain depend on the stationary distribution of the original chain, we can still exploit the quotient construction to deduce some relevant properties of the stationary distribution of the original chain. As one application, we shall establish inequalities that describe how fast the stationary distribution of Markov chains modeling evolutionary algorithms concentrates on the uniform populations as the mutation rate converges to 0. Further applications are discussed. One of the results related to the quotient construction method is a significant improvement of the corresponding result of the authors' previous conference paper [Mitavskiy et al. (2006) In: Simulated Evolution and Learning, Proceedings of SEAL 2006, Lecture Notes in Computer Science v. 4247, Springer Verlag, pp 726---733]. This papers implications are all strengthened accordingly.

Over the last decade the use of trans-dimensional sampling algorithms has become endemic in the statistical literature. In spite of their application however, there are few reliable methods to assess whether the underlying Markov chains have reached their stationary distribution. In this article we present a distance-based method for the comparison of trans-dimensional Markov chain sample output for a broad class of models. This diagnostic will simultaneously assess deviations between and within chains. Illustration of the analysis of Markov chain sample-paths is presented in simulated examples and in two common modelling situations: a finite mixture analysis and a change-point problem.

We consider the problem of giving explicit spectral bounds for time inhomogeneous Markov chains on a finite state space. We give bounds that apply when there exists a probability π such that each of the different steps corresponds to a nice ergodic Markov kernel with stationary measure π. For instance, our results provide sharp bounds for models such as semi-random transpositions and semi-random insertions (in these cases π is the uniform probability on the symmetric group).

This note is concerned with constructing time-homogeneous Markov chains on a countable state-space, including an absorbing state, such that the chain has a prescribed quasi-stationary distribution. The problem is characterized in terms of the underlying Q-matrix of the process. With no restriction on the Markov chain, it is straightforward to obtain a solution. For restricted processes this is no longer the case. We look in detail at population processes of birth–death and birth–catastrophe type, and in both cases obtain explicit constructions for Markov chains with any specified quasi-stationary distribution. The results are illustrated with examples.

The inhomogeneous Markov chain model is used to discriminate acceptor and donor sites in genomic DNA sequences. It outperforms statistical methods such as homogeneous Markov chain model, higher order Markov chain and interpolated Markov chain models, and machine-learning methods such as k-nearest neighbor and support vector machine as well. Besides its high accuracy, another advantage of inhomogeneous Markov chain model is its simplicity in computation. In the three states system (acceptor, donor, and neither), the inhomogeneous Markov chain model is combined with a three-layer feed forward neural network. Using this combined system 3175 primate splice-junction gene sequences have been tested, with a prediction accuracy of greater than 98%.

A gap in the author’s work on dynamic programming for ergodic control of partially observed Markov chains [V. S. Borkar, SIAM J. Control Optim., 39 (2000), pp. 673-681] is pointed out and a correction is provided.

International audience; Hidden Markov chains have been shown to be inadequate for data modeling under some complex conditions. In this work, we address the problem of statistical modeling of phenomena involving two heterogeneous system states. Such phenomena may arise in biology or communications, among other fields. Namely, we consider that a sequence of meaningful words is to be searched within a whole observation that also contains arbitrary one-by-one symbols. Moreover, a word may be interrupted at some site to be carried on later. Applying plain HMCs to such data, while ignoring their specificity, yields unsatisfactory results. The Phasic triplet Markov chain, proposed in this paper, overcomes this difficulty by means of an auxiliary underlying process in accordance with the triplet Markov chains theory. Related Bayesian restoration techniques and parameters estimation procedures according to the new model are then described. Finally, to assess the performance of the proposed model against the conventional hidden Markov chain model, experiments are conducted on synthetic and real data

In this paper, we formulate visual tracking as random walks on graph models with nodes representing superpixels and edges denoting relationships between superpixels. We integrate two novel graphs with the theory of Markov random walks, resulting in two Markov chains. First, an ergodic Markov chain is enforced to globally search for the candidate nodes with similar features to the template nodes. Second, an absorbing Markov chain is utilized to model the temporal coherence between consecutive frames. The final confidence map is generated by a structural model which combines both appearance similarity measurement derived by the random walks and internal spatial layout demonstrated by different target parts. The effectiveness of the proposed Markov chains as well as the structural model is evaluated both qualitatively and quantitatively. Experimental results on challenging sequences show that the proposed tracking algorithm performs favorably against state-of-the-art methods.

We propose a model of random walks on weighted graphs where the weights are interval valued, and connect it to reversible imprecise Markov chains. While the theory of imprecise Markov chains is now well established, this is a first attempt to model reversible chains. In contrast with the existing theory, the probability models that have to be considered are now non-convex. This presents a difficulty in computational sense, since convexity is critical for the existence of efficient optimization algorithms used in the existing models. The second part of the paper therefore addresses the computational issues of the model. The goal is finding sets of weights which maximize or minimize expectations corresponding to multiple steps transition probabilities. In particular, we present a local optimization algorithm and numerically test its efficiency. We show that its application allows finding close approximations of the globally best solutions in reasonable time., 17 pages, 1 algorithm, 1 table, 4 figures

Markov chains are commonly used in numerous signal processing and statistical modeling applications. This paper describes an approach to implement any first-order Markov chain using molecular reactions in general and DNA in particular. Markov chain consists of two parts: a set of states, and state transition probabilities. Each state is modeled by a unique molecular type, referred as a data molecule. Each state transition is modeled by a unique molecular type, referred as a control molecule, and a unique molecular reaction. Each reaction consumes data molecules of one state and produces data molecules of another state. The concentrations of control molecules are initialized according to the probabilities of corresponding state transitions in the chain. The steady-state probability of Markov chain is computed by equilibrium concentration of data molecules. We demonstrate our method for the Gambler's Ruin problem as an instance of the Markov chain process. Both stochastic chemical kinetics and mass-action kinetics validate the computed probabilities using the proposed model. The molecular reactions are then mapped to DNA strand displacement reactions. The error in the probability of ruin computed by the proposed model is shown to be less than 1% for DNA strand displacement reactions.

In this paper, we study a new way to define state aggregation of inhomogeneous Markov chains. This method consists in defining simulation morphisms between inhomogeneous Markov chains using their associated spaces of zero-charge measures and the evolution semigroups that arise from their transition matrices.

We search for the Markov chain with the optimal mixing rate where transitions are restricted to happen along a cycle of the states. We show that homogeneous, reversible chains are locally optimal for perturbations that make them inhomogeneous and non-reversible. Moreover, we show the optimality holds globally if only a single type of perturbation (either inhomogeneous or non-reversible) is applied. However, we conjecture global optimality holds for mixed perturbations as well, which is backed by simulation results. This paper complements previous results on bounds for mixing times of general Markov chains on the cycle [1].

In this paper, we study a model of quantum Markov chains that is a quantum analogue of Markov chains and is obtained by replacing probabilities in transition matrices with quantum operations. We show that this model is very suited to describe hybrid systems that consist of a quantum component and a classical one, although it has the same expressive power as another quantum Markov model proposed in the literature. Indeed, hybrid systems are often encountered in quantum information processing; for example, both quantum programs and quantum protocols can be regarded as hybrid systems. Thus, we further propose a model called hybrid quantum automata (HQA) that can be used to describe these hybrid systems that receive inputs (actions) from the outer world. We show the language equivalence problem of HQA is decidable in polynomial time. Furthermore, we apply this result to the trace equivalence problem of quantum Markov chains, and thus it is also decidable in polynomial time. Finally, we discuss model checking linear-time properties of quantum Markov chains, and show the quantitative analysis of regular safety properties can be addressed successfully., This paper has been accepted for publication in Information and Computation

We consider the problem of flexible modeling of higher order Markov chains when an upper bound on the order of the chain is known but the true order and nature of the serial dependence are unknown. We propose Bayesian nonparametric methodology based on conditional tensor factorizations, which can characterize any transition probability with a specified maximal order. The methodology selects the important lags and captures higher order interactions among the lags, while also facilitating calculation of Bayes factors for a variety of hypotheses of interest. We design efficient Markov chain Monte Carlo algorithms for posterior computation, allowing for uncertainty in the set of important lags to be included and in the nature and order of the serial dependence. The methods are illustrated using simulation experiments and real world applications. Supplementary materials for this article are available online.

In this paper we investigate the kernel estimator of the density for a stationary reversible Markov chain. The proofs are based on a new central limit theorem for a triangular array of reversible Markov chains obtained under conditions imposed to covariances, which has interest in itself.

For an irreducible continuous time Markov chain, we derive the distribution of the first passage time from a given state i to another given state j and the reversed passage time from j to i, each under the condition of no return to the starting point. When these two distributions are identical, we say that i and j are in time duality. We introduce a new condition called permuted balance that generalizes the concept of reversibility and provides sufficient criteria, based on the structure of the transition graph of the Markov chain. Illustrative examples are provided.

The authors present a new approach to find stochastic bounds for a Markov chain – namely with the use of the GPU for computing the bounds. A known algorithm [1, 2] is used and it is rewritten to suit the GPU architecture with the cooperation of the CPU. The authors do some experiments with matrices from various models as well as some random matrices. The tests are analyzed and some future considerations are given.

In the literature devoted to the efficient solution of Continuous Time Markov Chains (CTMCs) the notions of lump ability and reversibility have a central role. In the context of lump able Markov chains several definitions have been introduced: strong, exact and strict, just to mention a few of them. On the side of the analysis of reversible CTMCs the research community has shown great interest in the application of this notion with the aim of efficiently computing the stationary distribution of large models (e.g., obtained by composition of several processes). In this paper we show for the first time the relations between the above mentioned notions of lump ability and the concept of reversibility. The major outcome of our research is proving a strong connection between the notion of strict lump ability and that of reversibility.

In this work we develop a bootstrap method based on the theory of Markov chains. The method moves from the two competing objectives that a researcher pursues when performing a bootstrap procedure: (i) to preserve the structural similarity – in statistical sense – between the original and the bootstrapped sample; (ii) to assure a diversification of the latter with respect to the former. The original sample is assumed to be driven by a Markov chain. The approach we follow is to implement an optimization problem to estimate the memory of a Markov chain (i.e. its order) and to identify its relevant states. The basic ingredients of the model are the transition probabilities, whose distance is measured through a suitably defined functional. We apply the method to the series of electricity prices in Spain. A comparison with the Variable Length Markov Chain bootstrap, which is a well established bootstrap method, shows the superiority of our proposal in reproducing the dependence among data.

International audience; Given a finite Markov chain, we investigate the first minors of the transition matrix of a lifting of this Markov chain to covering trees. In a simple case we exhibit a nice factorisation of these minors, and we conjecture that it holds more generally.

In our monograph with Roynette and Yor (Najnudel et al., A Global View of Brownian Penalisations, MSJ Memoirs, vol. 19, Mathematical Society of Japan, Tokyo, 2009), we construct a σ-finite measure related to penalisations of different stochastic processes, including the Brownian motion in dimension 1 or 2, and a large class of linear diffusions. In the last chapter of the monograph, we define similar measures from recurrent Markov chains satisfying some technical conditions. In the present paper, we give a classification of these measures, in function of the minimal Martin boundary of the Markov chain considered at the beginning. We apply this classification to the examples considered at the end of Najnudel et al. (A Global View of Brownian Penalisations, MSJ Memoirs, vol. 19, Mathematical Society of Japan, Tokyo, 2009).

The basic building blocks of our models are counting processes, that is, nonnegative-integer-valued processes that are constant except for jumps of + 1. In this chapter, we introduce the notion of an intensity for a counting process and show how a counting process model can be specified by specifying a functional form for its intensity. The counting process corresponding to the intensity can be determined either as the solution of a stochastic equation or as the solution of a martingale problem. By writing a lattice-valued (e.g., \(\mathbb{Z}^{d}\)-valued) Markov chain in terms of counting processes that count the number of jumps of each of a countable number of types, we can specify the chain by specifying the intensities of the counting processes as functions of the state of the Markov chain.

This paper extends the Markov uni-chain series expansion theory to Markov multi-chains, i.e., to Markov chains having multiple ergodic classes and possible transient states. The introduced series expansion approximation (SEA) provides a controllable approximation for Markov multi-chain ergodic projectors which may be a useful tool in large-scale network analysis. As we will illustrate by means of numerical examples, the new algorithm is faster than the power algorithm for large networks.

In this paper, we established a representation for projector AA^D, based on which we developed two methods to compute Drazin inverse and discussed the technique for accelerating the convergence. At last, we applied them to the computation of Markov chains, which provided unconditioned and practical ways to the analysis of Markov chains.

A new object of the probability theory, a two-sided chain of events (symbols), is introduced. A theory of multi-steps Markov chains with long-range memory proposed in Phys. Rev. E, 68 (2003) 061107, is developed and used to establish the correspondence between these chains and two-sided sequences. The Markov chain is proved to be statistically equivalent to the definite two-sided one and vice versa. The results obtained for the binary chains are generalized to the chains of multi-valued symbols. Applicability of the two-sided chains to the examination of the one-dimensional Ising chains of classical spins is discussed.

Multi-dimensional asymptotically quasi-Toeplitz Markov chains with discrete and continuous time are introduced. Ergodicity and non-ergodicity conditions are proven. Numerically stable algorithm to calculate the stationary distribution is presented. An application of such chains in retrial queueing models with Batch Markovian Arrival Process is briefly illustrated.

We investigate the connections between extremal indices on the one hand and stability of Markov chains on the other hand. Both theories relate to the tail behaviour of stochastic processes, and we find a close link between the extremal index and geometric ergodicity. Our results are illustrated throughout with examples from simple MCMC chains.

Asymptotic stability of the optimal filter with respect to its initial conditions is investigated in this paper. Under the assumption that the observation function is one to one and the observation noise is sufficiently small, it is shown that exponential stability of the nonlinear filter holds for a large class of denumerable Markov chains, including all finite Markov chains. Throughout this paper, ergodicity of the signal process is not assumed.

In this paper, the equivalence between various divisible load-scheduling policies and continuous time Markov chains is demonstrated. This provides a basic unification of both data parallel divisible load scheduling and Markov chain models for the first time in 16 years of research. Such equivalence is demonstrated for divisible scheduling on linear daisy chains and single and two level tree networks.

Aggregated Markov processes related by similarity transformation are equivalent in that they cannot be distinguished by steady-state experiments. We derive an explicit formula for the set of all detailed-balance preserving similarity transformations between such continuous time Markov chains with N states. The matrices that define the allowed similarity transformations are found to be a simple non-linear function applied to almost any element of the special orthogonal group in N dimensions. Since a model is identifiable only if there is no similarity transformations to an equivalent model, we expect this result to prove useful in the theory of identification of aggregated Markov chains, an enterprise of growing importance as more and more single molecules yield to observation.

Markov chains are widely used to determine system performance and reliability characteristics. The vast majority of applications considers continuous-time Markov chains (CTMCs). This note motivates how concurrency theory can be extended (as opposed to twisted) to CTMCs. We provide the core motivation for the algebraic setup of Interactive Markov Chains. Therefore, this note should have better been baptized YIMC.

We consider how to determine all transition rates of an ionic channel when it can be conformationally described by a star-graph branch Markov chain with continuous time. It is found that all transition rates are uniquely determined by the distributions of their lifetime and death-time at the end state of each branch. An algorithm to exactly calculate all transition rates is developed. Numerical examples are included to demonstrate the application of our approach to data.

In this paper we discuss three important kinds of Markov chains used in Web search algorithms-the maximal irreducible Markov chain, the minimal irreducible Markov chain and the middle irreducible Markov chain. We discuss the stationary distributions, the convergence rates and the Maclaurin series of the stationary distributions of the three kinds of Markov chains. Among other things, our results show that the maximal and minimal Markov chains have the same stationary distribution and that the stationary distribution of the middle Markov chain reflects the real Web structure more objectively. Our results also prove that the maximal and middle Markov chains have the same convergence rate and that the maximal Markov chain converges faster than the minimal Markov chain when the damping factor $$ \alpha > \frac{1} {{{\sqrt 2 }}} $$ .

In an undergraduate course on stochastic processes, Markov chains are discussed in great detail. Textbooks on stochastic processes provide interesting properties of finite Markov chains. This note discusses one such property regarding the number of steps in which a state is reachable or accessible from another state in a finite Markov chain with M (≥ 2) states.

In this paper, we study several linear-time equivalences (Markovian trace equivalence, failure and ready trace equivalence) for continuous-time Markov chains that refer to the probabilities for timed execution paths. Our focus is on testing scenarios by means of push-button experiments with appropriate trace machines and a discussion of the connections between the equivalences. For Markovian trace equivalence, we provide alternative characterizations, including one that abstracts away from the time instances where actions are observed, but just reports on the average sojourn times in the states. This result is used for a reduction of the question whether two finite-state continuous-time Markov chains are Markovian trace equivalent to the probabilistic trace equivalence problem for discrete-time Markov chains (and the latter is known to be solvable in polynomial time).

This paper deals with Blackwell optimality for continuous-time controlled Markov chains with compact Borel action space, and possibly unbounded reward (or cost) rates and unbounded transition rates. We prove the existence of a deterministic stationary policy which is Blackwell optimal in the class of all admissible (nonstationary) Markov policies, thus extending previous results that analyzed Blackwell optimality in the class of stationary policies. We compare our assumptions to the corresponding ones for discrete-time Markov controlled processes.

The random transition count for Markov chains with a fixed length is not a complete statistic for some natural classes of Markov chains. But this statistic is always complete for “Markov bridge”, when the initial and the final state are fixed.

Monte Carlo algorithms often depend on Markov chains to sample from very large data sets. A key ingredient in the design of an efficient Markov chain is determining rigorous bounds on how quickly the chain "mixes," or converges, to its stationary distribution. This survey provides an overview of several useful techniques

A Viterbi path of length n of a discrete Markov chain is a sequence of n+1 states that has the greatest probability of ocurring in the Markov chain. We divide the space of all Markov chains into Viterbi regions in which two Markov chains are in the same region if they have the same set of Viterbi paths. The Viterbi paths of regions of positive measure are called Viterbi sequences. Our main results are (1) each Viterbi sequence can be divided into a prefix, periodic interior, and suffix, and (2) as n increases to infinity (and the number of states remains fixed), the number of Viterbi regions remains bounded. The Viterbi regions correspond to the vertices of a Newton polytope of a polynomial whose terms are the probabilities of sequences of length n. We characterize Viterbi sequences and polytopes for two- and three-state Markov chains., Comment: 15 pages, 2 figures, to appear in Journal of Symbolic Computation

This paper proposes a Markov chain method to predict the growth dynamics of the individual nodes in scale-free networks, and uses this to calculate numerically the degree distribution. We first find that the degree evolution of a node in the BA model is a nonhomogeneous Markov chain. An efficient algorithm to calculate the degree distribution is developed by the theory of Markov chains. The numerical results for the BA model are consistent with those of the analytical approach. A directed network with the logarithmic growth is introduced. The algorithm is applied to calculate the degree distribution for the model. The numerical results show that the system self-organizes into a scale-free network.

In this paper, by using theories of stochastic process and computer algebra, a reliable algorithm for computing the high-order transition probability matrix of finite Markov chain is established, a new mechanical procedure markovproc is established, too. The procedure markovproc give not only the exact expression of the Markov chains, but also the limiting probability. Some examples are presented to illustrate the implementation of the algorithm.