Search

We develop a computationally efficient learning-based forward-backward stochastic differential equations (FBSDE) controller for both continuous and hybrid dynamical (HD) systems subject to stochastic noise and state constraints. Solutions to stochastic optimal control (SOC) problems satisfy the Hamilton-Jacobi-Bellman (HJB) equation. Using current FBSDE-based solutions, the optimal control can be obtained from the HJB equations using deep neural networks (e.g., long short-term memory (LSTM) networks). To ensure the learned controller respects the constraint boundaries, we enforce the state constraints using a soft penalty function. In addition to previous works, we adapt the deep FBSDE (DFBSDE) control framework to handle HD systems consisting of continuous dynamics and a deterministic discrete state change. We demonstrate our proposed algorithm in simulation on a continuous nonlinear system (cart-pole) and a hybrid nonlinear system (five-link biped).

In this article we analyze a partial-information Nash Q-learning algorithm for a general 2-player stochastic game. Partial information refers to the setting where a player does not know the strategy or the actions taken by the opposing player. We prove convergence of this partially informed algorithm for general 2-player games with finitely many states and actions, and we confirm that the limiting strategy is in fact a full-information Nash equilibrium. In implementation, partial information offers simplicity because it avoids computation of Nash equilibria at every time step. In contrast, full-information Q-learning uses the Lemke-Howson algorithm to compute Nash equilibria at every time step, which can be an effective approach but requires several assumptions to prove convergence and may have runtime error if Lemke-Howson encounters degeneracy. In simulations, the partial information results we obtain are comparable to those for full-information Q-learning and fictitious play.

We analyze a fixed-point algorithm for reinforcement learning (RL) of optimal portfolio mean-variance preferences in the setting of multivariate generalized autoregressive conditional-heteroskedasticity (MGARCH) with a small penalty on trading. A numerical solution is obtained using a neural network (NN) architecture within a recursive RL loop. A fixed-point theorem proves that NN approximation error has a big-oh bound that we can reduce by increasing the number of NN parameters. The functional form of the trading penalty has a parameter $\epsilon>0$ that controls the magnitude of transaction costs. When $\epsilon$ is small, we can implement an NN algorithm based on the expansion of the solution in powers of $\epsilon$. This expansion has a base term equal to a myopic solution with an explicit form, and a first-order correction term that we compute in the RL loop. Our expansion-based algorithm is stable, allows for fast computation, and outputs a solution that shows positive testing performance.

In this paper, we propose a new methodology for state constrained stochastic optimal control (SOC) problems. The solution is based on past work in solving SOC problems using forward-backward stochastic differential equations (FBSDE). Our approach in solving the FBSDE utilizes a deep neural network (DNN), specifically Long Short-Term Memory (LSTM) networks. LSTMs are chosen to solve the FBSDE to address the curse of dimensionality, non-linearities, and long time horizons. In addition, the state constraints are incorporated using a hard penalty function, resulting in a controller that respects the constraint boundaries. Numerical instability that would be introduced by the penalty function is dealt with through an adaptive update scheme. The control design methodology is applicable to a large class of control problems. The performance and scalability of our proposed algorithm are demonstrated by numerical simulations., Comment: 6 pages, 5 figures, American Control Conference 2021

This article explores the relationship between the SPX and VIX options markets. High-strike VIX call options are used to hedge tail risk in the SPX, which means that SPX options are a reflection of the extreme-strike asymptotics of VIX options, and vice versa. This relationship can be quantified using moment formulas in a model-free way. Comparisons are made between VIX and SPX implied volatilities along with various examples of stochastic volatility models., Comment: Special Thank You to Roger Lee for your help in this paper

Principal component analysis (PCA) is a useful tool when trying to construct factor models from historical asset returns. For the implied volatilities of U.S. equities there is a PCA-based model with a principal eigenportfolio whose return time series lies close to that of an overarching market factor. The authors show that this market factor is the index resulting from the daily compounding of a weighted average of implied-volatility returns, with weights based on the options' open interest (OI) and Vega. The authors also analyze the singular vectors derived from the tensor structure of the implied volatilities of S&P500 constituents, and find evidence indicating that some type of OI and Vega-weighted index should be one of at least two significant factors in this market.

We present an expansion for portfolio optimization in the presence of small, instantaneous, quadratic transaction costs. Specifically, the magnitude of transaction costs has a coefficient that is of the order $\epsilon$ small, which leads to the optimization problem having an asymptotically-singular Hamilton-Jacobi-Bellman equation whose solution can be expanded in powers of $\sqrt\epsilon$. In this paper we derive explicit formulae for the first two terms of this expansion. Analysis and simulation are provided to show the behavior of this approximating solution.

This paper shows how to recover a stochastic volatility model (SVM) from a market model of the VIX futures term structure. Market models have more flexibility for fitting of curves than do SVMs, and therefore are better suited for pricing VIX futures and VIX derivatives. But the VIX itself is a derivative of the S&P500 (SPX) and it is common practice to price SPX derivatives using an SVM. Therefore, consistent modeling for both SPX and VIX should involve an SVM that can be obtained by inverting the market model. This paper's main result is a method for the recovery of a stochastic volatility function by solving an inverse problem where the input is the VIX function given by a market model. Analysis will show conditions necessary for there to be a unique solution to this inverse problem. The models are consistent if the recovered volatility function is non-negative. Examples are presented to illustrate the theory, to highlight the issue of negativity in solutions, and to show the potential for inconsistency in non-Markov settings.

This paper considers a non-Markov control problem arising in a financial market where asset returns depend on hidden factors. The problem is non-Markov because nonlinear filtering is required to make inference on these factors, and hence the associated dynamic program effectively takes the filtering distribution as one of its state variables. This is of significant difficulty because the filtering distribution is a stochastic probability measure of infinite dimension, and therefore the dynamic program has a state that cannot be differentiated in the traditional sense. This lack of differentiability means that the problem cannot be solved using a Hamilton-Jacobi-Bellman (HJB) equation. This paper will show how the problem can be analyzed and solved using backward stochastic differential equations (BSDEs), with a key tool being the problem's dual formulation., Comment: Part of this research was performed while the author was visiting the Institute for Pure and Applied Mathematics (IPAM), which is supported by the National Science Foundation, Mathematical Finance (2018)

We consider partially observed multiscale diffusion models that are specified up to an unknown vector parameter. We establish for a very general class of test functions that the filter of the original model converges to a filter of reduced dimension. Then, this result is used to justify statistical estimation for the unknown parameters of interest based on the model of reduced dimension but using the original available data. This allows to learn the unknown parameters of interest while working in lower dimensions, as opposed to working with the original high dimensional system. Simulation studies support and illustrate the theoretical results., Comment: SIAM Journal of Uncertainty Quantification, 2017

These are course notes on the application of SDEs to options pricing. The author was partially supported by NSF grant DMS-0739195., Comment: These are an evolving set of course notes. Eventually I hope to make them a book. They are posted on the arXiv so that others may see my approach to the topic

These notes were originally written for the Stochastic Analysis Seminar in the Department of Operations Research and Financial Engineering at Princeton University, in February of 2011. The seminar was attended and supported by members of the Research Training Group, with the author being partially supported by NSF grant DMS-0739195., Comment: 94 pages

Filtering and parameter estimation under partial information for multiscale problems is studied in this paper. After proving mean square convergence of the nonlinear filter to a filter of reduced dimension, we establish that the conditional (on the observations) log-likelihood process has a correction term given by a type of central limit theorem. To achieve this we assume that the operator of the (hidden) fast process has a discrete spectrum and an orthonormal basis of eigenfunctions. Based on these results, we then propose to estimate the unknown parameters of the model based on the limiting log-likelihood, which is an easier function to optimize because it of reduced dimension. We also establish consistency and asymptotic normality of the maximum likelihood estimator based on the reduced log-likelihood. Simulation results illustrate our theoretical findings., Comment: Keywords: Ergodic filtering, fast mean reversion, homogenization, Zakai equation, maximum likelihood estimation, central limit theory

We consider filtering for a hidden Markov model that evolves with multiple time scales in the hidden states. In particular, we consider the case where one of the states is a scaled Ornstein-Uhlenbeck process with fast reversion to a shifting-mean that is controlled by a continuous time Markov chain modeling regime change. We show that the nonlinear filter for such a process can be approximated by an averaged filter that asymptotically coincides with the true nonlinear filter of the regime-changing Markov chain as the rate of mean reversion approaches infinity. The asymptotics exploit weak converge of the state variables to an invariant distribution, which is significantly different from the strong convergence used to obtain asymptotic results in "Filtering for Fast Mean-Reverting Processes" (19).

We formulate and analyze an inverse problem using derivatives prices to obtain an implied filtering density on volatility's hidden state. Stochastic volatility is the unobserved state in a hidden Markov model (HMM) and can be tracked using Bayesian filtering. However, derivative data can be considered as conditional expectations that are already observed in the market, and which can be used as input to an inverse problem whose solution is an implied conditional density on volatility. Our analysis relies on a specification of the martingale change of measure, which we refer to as \textit{separability}. This specification has a multiplicative component that behaves like a risk premium on volatility uncertainty in the market. When applied to SPX options data, the estimated model and implied densities produce variance-swap rates that are consistent with the VIX volatility index. The implied densities are relatively stable over time and pick up some of the monthly effects that occur due to the options' expiration, indicating that the volatility-uncertainty premium could experience cyclic effects due to the maturity date of the options.