Search

This work is concerned with optimal harvesting problems in random environments. In contrast to the existing literature, the Markov chain is hidden and can only be observed in a Gaussian white noise in our work. We first use the Wonham filter to estimate the Markov chain from the observable evolution of the given process so as to convert the original problem to a completely observable one. Then we treat the resulting optimal control problem. Because the problem is virtually impossible to solve in closed form, our main effort is devoted to developing numerical approximation algorithms. To approximate the value function and optimal strategies, Markov chain approximation methods are used to construct a discrete-time controlled Markov chain. Convergence of the algorithm is proved by weak convergence method and suitable scaling. A numerical example is provided to demonstrate the results. [ABSTRACT FROM AUTHOR]

Abstract: Facing the pressing needs of many applications in network and control systems, this paper introduces a class of nonlinear systems with random time delays and derives conditions on moment exponential stability of the underlying systems. The system model is versatile and can accommodate a wide variety of situations. The stability analysis to date in the literature is mostly delay independent. To highlight the role of random delay for stability, this paper focuses on delay-dependent stability. Dependence of stability on random time delays introduces technical difficulties beyond the existing literature. We model the random time delays by a continuous-time Markov chain involving two-time scales defined by a small parameter . leading to a two-time scale framework. The random delays change their values with a fast varying mode and a slowly evolving effect. Under broad conditions, the stability of the system is studied using a limit system in the sense of weak convergence of probability measures. Using the limit system as a bridge, this paper establishes the Razumikhin-type criteria on the moment exponential stability. These criteria show that the mean of the random time delay with respect to the stationary distribution of the fast changing part of the Markov chain plays an important role in the moment exponential stability, which presents a novel feature of our work. In particular, we show that the overall system may be stabilized by the Markov switching even when some of the underlying subsystems are unstable, which shows that the Markov chain may serve as a stabilization factor. Explicit conditions for moment exponential stability are derived when the system is linear. Examples are given to illustrate our results. [Copyright &y& Elsevier]

Abstract: This paper develops numerical methods for finding optimal dividend pay-out and reinsurance policies. A generalized singular control formulation of surplus and discounted payoff function is introduced, where the surplus is modeled by a regime-switching process subject to both regular and singular controls. To approximate the value function and optimal controls, Markov chain approximation techniques are used to construct a discrete-time controlled Markov chain. The proofs of the convergence of the approximation sequence to the surplus process and the value function are given. Examples of proportional and excess-of-loss reinsurance are presented to illustrate the applicability of numerical methods. [Copyright &y& Elsevier]

Abstract: This work develops numerical approximation algorithms for solutions of stochastic differential equations with Markovian switching. The existing numerical algorithms all use a discrete-time Markov chain for the approximation of the continuous-time Markov chain. In contrast, we generate the continuous-time Markov chain directly, and then use its skeleton process in the approximation algorithm. Focusing on weak approximation, we take a re-embedding approach, and define the approximation and the solution to the switching stochastic differential equation on the same space. In our approximation, we use a sequence of independent and identically distributed (i.i.d.) random variables in lieu of the common practice of using Brownian increments. By virtue of the strong invariance principle, we ascertain rates of convergence in the pathwise sense for the weak approximation scheme. [Copyright &y& Elsevier]

Abstract: This paper is concerned with asymptotic properties of consensus-type algorithms for networked systems whose topologies switch randomly. The regime-switching process is modeled as a discrete-time Markov chain with a finite state space. The consensus control is achieved by using stochastic approximation methods. In the setup, the regime-switching process (the Markov chain) contains a rate parameter in the transition probability matrix that characterizes how frequently the topology switches. On the other hand, the consensus control algorithm uses a stepsize that defines how fast the network states are updated. Depending on their relative values, three distinct scenarios emerge. Under suitable conditions, we show that when , a continuous-time interpolation of the iterates converges weakly to a system of randomly switching ordinary differential equations modulated by a continuous-time Markov chain. In this case a scaled sequence of tracking errors converges to a system of switching diffusion. When , the network topology is almost non-switching during consensus control transient intervals, and hence the limit dynamic system is simply an autonomous differential equation. When , the Markov chain acts as a fast varying noise, and only its averaged network matrices are relevant, resulting in a limit differential equation that is an average with respect to the stationary measure of the Markov chain. Simulation results are presented to demonstrate these findings. [Copyright &y& Elsevier]

This work develops an approximation procedure to find optimal annuity-purchasing strategies for minimizing the probability of lifetime ruin. The wealth is modelled as a regime-switching diffusion modulated by a continuous-time Markov chain. Based on Markov chain approximation techniques, a discrete-time controlled Markov chain with two components is constructed. Under simple conditions, the convergence of the approximation sequence to the wealth process is obtained. The convergence of the approximation to the value function is also established. Several examples are provided to demonstrate the performance of the algorithms. [ABSTRACT FROM AUTHOR]

This work is concerned with pricing American fixed lookback put options. The underlying asset is modeled as a switching diffusion process, where the switching is represented by a continuous-time Markov chain. The switching diffusion delineates stochastic volatility effectively. Nevertheless, this formulation together with the lookback style put option makes it virtually impossible to find closed-form solutions. As a viable alternative, a stochastic approximation algorithm is suggested. The convergence and rates of convergence of the algorithm are established. [ABSTRACT FROM AUTHOR]

This work is concerned with weak convergence of non-Markov random processes modulated by a Markov chain. The motivation of our study stems from a wide variety of applications in actuarial science, communication networks, production planning, manufacturing and financial engineering. Owing to various modelling considerations, the modulating Markov chain often has a large state space. Aiming at reduction of computational complexity, a two-time-scale formulation is used. Under this setup, the Markov chain belongs to the class of nearly completely decomposable class, where the state space is split into several subspaces. Within each subspace, the transitions of the Markov chain varies rapidly, and among different subspaces, the Markov chain moves relatively infrequently. Aggregating all the states of the Markov chain in each subspace to a single super state leads to a new process. It is shown that under such aggregation schemes, a suitably scaled random sequence converges to a switching diffusion process. [ABSTRACT FROM AUTHOR]

Focusing on hybrid diffusions in which continuous dynamics and discrete events coexist, this work is concerned with approximation of solutions for hybrid stochastic differential equations with a state-dependent switching process. Iterative algorithms are developed. The continuous-state-dependent switching process presents added difficulties in analyzing the numerical procedures. Weak convergence of the algorithms is established by a martingale problem formulation first. This weak convergence result is then used as a bridge to obtain strong convergence. In this process, the existence and uniqueness of the solution of the switching diffusions with continuous-state-dependent switching are obtained. In contrast to existing results of solutions of stochastic differential equations in which the Picard iterations are utilized, Euler's numerical schemes are considered here. Moreover, decreasing-stepsize algorithms together with their weak convergence are given. Numerical experiments are also provided for demonstration. [ABSTRACT FROM AUTHOR]

Abstract: This work develops an approximation procedure for portfolio selection with bounded constraints. Based on the Markov chain approximation techniques, numerical procedures are constructed for the utility optimization task. Under simple conditions, the convergence of the approximation sequences to the wealth process and the optimal utility function is established. Numerical examples are provided to illustrate the performance of the algorithms. [Copyright &y& Elsevier]

Stochastic optimization/approximation algorithms are widely used to recursively estimate the optimum of a suitable function or its root under noisy observations when this optimum or root is a constant or evolves randomly according to slowly time-varying continuous sample paths. In comparison, this paper analyzes the asymptotic properties of stochastic optimization/approximation algorithms for recursively estimating the optimum or root when it evolves rapidly with nonsmooth (jump-changing) sample paths. The resulting problem falls into the category of regime-switching stochastic approximation algorithms with two-time scales. Motivated by emerging applications in wireless communications, and system identification, we analyze asymptotic behavior of such algorithms. Our analysis assumes that the noisy observations contain a (nonsmooth) jump process modeled by a discrete-time Markov chain whose transition frequency varies much faster than the adaptation rate of the stochastic optimization algorithm. Using stochastic averaging, we prove convergence of the algorithm. Rate of convergence of the algorithm is obtained via bounds on the estimation errors and diffusion approximations. Remarks on improving the convergence rates through iterate averaging, and limit mean dynamics represented by differential inclusions are also presented. [ABSTRACT FROM AUTHOR]

This work is concerned with rates of convergence of Markov chain approximation methods for controlled switching diffusions. The cost function is defined on an infinite horizon with stopping times and without discount. Displaying both continuous dynamics and discrete events, the discrete events are modeled by continuous-time Markov chains to delineate a random environment and other random factors that cannot be represented by diffusion processes. This paper presents a first attempt using a probabilistic approach to treat such rates of convergence problems. In addition, in contrast to the significant developments in the literature using partial differential equation (PDE) methods for the approximation of controlled diffusions, there do not yet appear to be any PDE results to date for rates of convergence of numerical solutions for controlled switching diffusions, to the best of our knowledge. Although some of the working conditions in this paper such as the one-dimensional continuous state variable, nondegenerate diffusions, and control only on the drift may be seemingly strong, they are adequate as the starting point for using this new approach to treat the rates of convergence problems. Moreover, in the literature, to prove the convergence using Markov chain approximation methods for control problems involving cost functions with stopping (even for uncontrolled diffusion without switching), an added assumption was used to avoid the socalled tangency problem. As a by-product of our approach, by modifying the value function, it is demonstrated that the anticipated tangency problem will not arise in the sense of convergence in probability and in the sense of L1. [ABSTRACT FROM AUTHOR]

This article is concerned with a numerical method using stochastic approximation approach for an optimal trading (buy and sell) strategy. The underlying asset price is governed by a mean-reverting stochastic process. The objective is to buy and sell the asset so as to maximize an overall expected return. One of the advantages of our approach is that the underlying asset is model free. Only mean reversion is required. Slippage cost is imposed on each transaction. Convergence of the algorithms is provided. Numerical examples are reported to demonstrate the results. [ABSTRACT FROM AUTHOR]

Motivated by an important problem of load balancing in parallel computing, this paper examines a modified algorithm to enhance Q-learning methods, especially in asynchronous recursive procedures for self-adaptive load distribution at runtime. Unlike the existing projection method that utilizes a fixed region, our algorithm employs a sequence of growing truncation bounds to ensure the boundedness of the iterates. Convergence and rates of convergence of the proposed algorithm are established. This class of algorithms has broad applications in signal processing, learning, financial engineering, and other related fields. [ABSTRACT FROM AUTHOR]

This work is concerned with option pricing. Stochastic approximation/optimization algorithms are proposed and analyzed. The underlying stock price evolves according to two geometric Brownian motions coupled by a continuous-time finite state Markov chain. Recursive stochastic approximation algorithms are developed to estimate the implied volatility. Convergence of the algorithm is proved. Rate of convergence is also ascertained. Then real market data are used to compare our algorithms with other schemes. [ABSTRACT FROM AUTHOR]

This paper analyzes the tracking properties of the least mean squares (LMS) algorithm when the underlying parameter evolves according to a finite-state Markov chain with infrequent jumps. First, using perturbed Liapunov function methods, mean-square error estimates are obtained for the tracking error. Then using recent results on two-time-scale Markov chains, mean ordinary differential equation and diffusion approximation results are obtained. It is shown that a sequence of the centered tracking errors converges to an ordinary differential equation. Moreover, a suitably scaled sequence of the tracking errors converges weakly to a diffusion process. It is also shown that iterate averaging of the tracking algorithm results in optimal asymptotic convergence rate in an appropriate sense. Two application examples, analysis of the performance of an adaptive multiuser detection algorithm in a direct-sequence code-division multiple-access (DS/CDMA) system, and tracking analysis of the state of a hidden Markov model (HMM) with infrequent jumps, are presented. [ABSTRACT FROM AUTHOR]

Aiming at reduction of complexity, this work is concerned with two-time-scale Markov chains and applications to quasi-birth-death queues. Asymptotic expansions of probability vectors are constructed and justified. Lumping all states of the Markov chain in each subspace into a single state, an aggregated process is shown to converge to a continuous-time Markov chain whose generator is an average with respect to the stationary measures. Then a suitably scaled sequence is shown to converge to a switching diffusion process. Extensions of the results are presented together with examples of quasi-birth-death queues. [ABSTRACT FROM AUTHOR]

This work is concerned with diffusion processes having fast and slow components. It was known that under suitable assumptions the slow component can be approximated by the Markov process with averaged characteristics. In this work, asymptotic expansions for the solutions of the Kolmogorov backward equations are constructed and justified. Certain probabilistic conclusions and examples are also provided. [ABSTRACT FROM AUTHOR]

Abstract: This work develops numerical approximation methods for quantile hedging involving mortality components for contingent claims in incomplete markets, in which guaranteed minimum death benefits (GMDBs) could not be perfectly hedged. A regime-switching jump-diffusion model is used to delineate the dynamic system and the hedging function for GMDBs, where the switching is represented by a continuous-time Markov chain. Using Markov chain approximation techniques, a discrete-time controlled Markov chain with two component is constructed. Under simple conditions, the convergence of the approximation to the value function is established. Examples of quantile hedging model for guaranteed minimum death benefits under linear jumps and general jumps are also presented. [ABSTRACT FROM AUTHOR]

It is widely recognized that unregulated harvesting and hunting of biological resources can be harmful and endanger ecosystems. Therefore, various measures to prevent the biological resources from destruction and to protect the ecological environment have been taken. An effective resolution is to designate protection zones where harvesting and hunting are prohibited. Assuming that migration can occur between protected and unprotected areas, a fundamental question is, how large should a protection zone be so that the species in both the protection subregion and natural environment are able to survive. Devoted to answering the question, this paper aims at studying ecosystems that are subject to random noise represented by Brownian motion. Sufficient conditions for permanence and extinction are obtained, which are sharp and close to necessary conditions. Moreover, ergodicity, convergence of probability measures to that of the invariant measure under total variation norm, and rates of convergence are obtained. [ABSTRACT FROM AUTHOR]

Focusing on asymptotic behavior of a stochastic SIR epidemic model represented by a system of stochastic differential equations with a degenerate diffusion, this paper provides suficient conditions that are very close to the necessary ones for the permanence. In addition, this paper develops ergodicity of the underlying system. It is proved that the transition probabilities converge in total variation norm to the invariant measure. Our result gives a precise characterization of the support of the invariant measure. Rates of convergence are also ascertained. It is shown that the rate is not too far from exponential in that the convergence speed is of the form of a polynomial of any degree. [ABSTRACT FROM AUTHOR]