Search

In a reinforcement learning (RL) setting, the agent's optimal strategy heavily depends on her risk preferences and the underlying model dynamics of the training environment. These two aspects influence the agent's ability to make well-informed and time-consistent decisions when facing testing environments. In this work, we devise a framework to solve robust risk-aware RL problems where we simultaneously account for environmental uncertainty and risk with a class of dynamic robust distortion risk measures. Robustness is introduced by considering all models within a Wasserstein ball around a reference model. We estimate such dynamic robust risk measures using neural networks by making use of strictly consistent scoring functions, derive policy gradient formulae using the quantile representation of distortion risk measures, and construct an actor-critic algorithm to solve this class of robust risk-aware RL problems. We demonstrate the performance of our algorithm on a portfolio allocation example., Comment: 29 pages, 3 figures

We study the perfect information Nash equilibrium between a broker and her clients -- an informed trader and an uniformed trader. In our model, the broker trades in the lit exchange where trades have instantaneous and transient price impact with exponential resilience, while both clients trade with the broker. The informed trader and the broker maximise expected wealth subject to inventory penalties, while the uninformed trader is not strategic and sends the broker random buy and sell orders. We characterise the Nash equilibrium of the trading strategies with the solution to a coupled system of forward-backward stochastic differential equations (FBSDEs). We solve this system explicitly and study the effect of information, profitability, and inventory control in the trading strategies of the broker and the informed trader., Comment: 24 pages, 3 figures

We consider the problem where an agent aims to combine the views and insights of different experts' models. Specifically, each expert proposes a diffusion process over a finite time horizon. The agent then combines the experts' models by minimising the weighted Kullback-Leibler divergence to each of the experts' models. We show existence and uniqueness of the barycentre model and proof an explicit representation of the Radon-Nikodym derivative relative to the average drift model. We further allow the agent to include their own constraints, which results in an optimal model that can be seen as a distortion of the experts' barycentre model to incorporate the agent's constraints. Two deep learning algorithms are proposed to find the optimal drift of the combined model, allowing for efficient simulations. The first algorithm aims at learning the optimal drift by matching the change of measure, whereas the second algorithm leverages the notion of elicitability to directly estimate the value function. The paper concludes with a extended application to combine implied volatility smiles models that were estimated on different datasets.

We investigate sample-based learning of conditional distributions on multi-dimensional unit boxes, allowing for different dimensions of the feature and target spaces. Our approach involves clustering data near varying query points in the feature space to create empirical measures in the target space. We employ two distinct clustering schemes: one based on a fixed-radius ball and the other on nearest neighbors. We establish upper bounds for the convergence rates of both methods and, from these bounds, deduce optimal configurations for the radius and the number of neighbors. We propose to incorporate the nearest neighbors method into neural network training, as our empirical analysis indicates it has better performance in practice. For efficiency, our training process utilizes approximate nearest neighbors search with random binary space partitioning. Additionally, we employ the Sinkhorn algorithm and a sparsity-enforced transport plan. Our empirical findings demonstrate that, with a suitably designed structure, the neural network has the ability to adapt to a suitable level of Lipschitz continuity locally. For reproducibility, our code is available at \url{https://github.com/zcheng-a/LCD_kNN}.

When an investor is faced with the option to purchase additional information regarding an asset price, how much should she pay? To address this question, we solve for the indifference price of information in a setting where a trader maximizes her expected utility of terminal wealth over a finite time horizon. If she does not purchase the information, then she solves a partial information stochastic control problem, while, if she does purchase the information, then she pays a cost and receives partial information about the asset's trajectory. We further demonstrate that when the investor can purchase the information at any stopping time prior to the end of the trading horizon, she chooses to do so at a deterministic time.

One approach to reducing greenhouse gas (GHG) emissions is to incentivize carbon capturing and carbon reducing projects while simultaneously penalising excess GHG output. In this work, we present a novel market framework and characterise the optimal behaviour of GHG offset credit (OC) market participants in both single-player and two-player settings. The single player setting is posed as an optimal stopping and control problem, while the two-player setting is posed as optimal stopping and mixed-Nash equilibria problem. We demonstrate the importance of acting optimally using numerical solutions and Monte Carlo simulations and explore the differences between the homogeneous and heterogeneous players. In both settings, we find that market participants benefit from optimal OC trading and OC generation.

We study a reinsurer who faces multiple sources of model uncertainty. The reinsurer offers contracts to $n$ insurers whose claims follow compound Poisson processes representing both idiosyncratic and systemic sources of loss. As the reinsurer is uncertain about the insurers' claim severity distributions and frequencies, they design reinsurance contracts that maximise their expected wealth subject to an entropy penalty. Insurers meanwhile seek to maximise their expected utility without ambiguity. We solve this continuous-time Stackelberg game for general reinsurance contracts and find that the reinsurer prices under a distortion of the barycentre of the insurers' models. We apply our results to proportional reinsurance and excess-of-loss reinsurance contracts, and illustrate the solutions numerically. Furthermore, we solve the related problem where the reinsurer maximises, still under ambiguity, their expected utility and compare the solutions.

This paper proposes a novel framework for identifying an agent's risk aversion using interactive questioning. Our study is conducted in two scenarios: a one-period case and an infinite horizon case. In the one-period case, we assume that the agent's risk aversion is characterized by a cost function of the state and a distortion risk measure. In the infinite horizon case, we model risk aversion with an additional component, a discount factor. Assuming the access to a finite set of candidates containing the agent's true risk aversion, we show that asking the agent to demonstrate her optimal policies in various environment, which may depend on their previous answers, is an effective means of identifying the agent's risk aversion. Specifically, we prove that the agent's risk aversion can be identified as the number of questions tends to infinity, and the questions are randomly designed. We also develop an algorithm for designing optimal questions and provide empirical evidence that our method learns risk aversion significantly faster than randomly designed questions in simulations. Our framework has important applications in robo-advising and provides a new approach for identifying an agent's risk preferences.

In the Milky Way, the distribution of stars in the $[\alpha/\mathrm{Fe}]$ vs. $[\mathrm{Fe/H}]$ and $[\mathrm{Fe/H}]$ vs. age planes holds essential information about the history of star formation, accretion, and dynamical evolution of the Galactic disk. We investigate these planes by applying novel statistical methods called copulas and elicitable maps to the ages and abundances of red giants in the APOGEE survey. We find that the low- and high-$\alpha$ disk stars have a clean separation in copula space and use this to provide an automated separation of the $\alpha$ sequences using a purely statistical approach. This separation reveals that the high-$\alpha$ disk ends at the same [$\alpha$/Fe] and age at high $[\mathrm{Fe/H}]$ as the low-$[\mathrm{Fe/H}]$ start of the low-$\alpha$ disk, thus supporting a sequential formation scenario for the high- and low-$\alpha$ disks. We then combine copulas with elicitable maps to precisely obtain the correlation between stellar age $\tau$ and metallicity $[\mathrm{Fe/H}]$ conditional on Galactocentric radius $R$ and height $z$ in the range $0 < R < 20$ kpc and $|z| < 2$ kpc. The resulting trends in the age-metallicity correlation with radius, height, and [$\alpha$/Fe] demonstrate a $\approx 0$ correlation wherever kinematically-cold orbits dominate, while the naively-expected negative correlation is present where kinematically-hot orbits dominate. This is consistent with the effects of spiral-driven radial migration, which must be strong enough to completely flatten the age-metallicity structure of the low-$\alpha$ disk., Comment: 18 pages, 16 figures, Submitted to MNRAS

We define and develop an approach for risk budgeting allocation - a risk diversification portfolio strategy - where risk is measured using a dynamic time-consistent risk measure. For this, we introduce a notion of dynamic risk contributions that generalise the classical Euler contributions and which allow us to obtain dynamic risk contributions in a recursive manner. We prove that, for the class of coherent dynamic distortion risk measures, the risk allocation problem may be recast as a sequence of strictly convex optimisation problems. Moreover, we show that self-financing dynamic risk budgeting strategies with initial wealth of 1 are scaled versions of the solution of the sequence of convex optimisation problems. Furthermore, we develop an actor-critic approach, leveraging the elicitability of dynamic risk measures, to solve for risk budgeting strategies using deep learning.

This article explores the optimisation of trading strategies in Constant Function Market Makers (CFMMs) and centralised exchanges. We develop a model that accounts for the interaction between these two markets, estimating the conditional dependence between variables using the concept of conditional elicitability. Furthermore, we pose an optimal execution problem where the agent hides their orders by controlling the rate at which they trade. We do so without approximating the market dynamics. The resulting dynamic programming equation is not analytically tractable, therefore, we employ the deep Galerkin method to solve it. Finally, we conduct numerical experiments and illustrate that the optimal strategy is not prone to price slippage and outperforms na\"ive strategies.

The objectives of option hedging/trading extend beyond mere protection against downside risks, with a desire to seek gains also driving agent's strategies. In this study, we showcase the potential of robust risk-aware reinforcement learning (RL) in mitigating the risks associated with path-dependent financial derivatives. We accomplish this by leveraging a policy gradient approach that optimises robust risk-aware performance criteria. We specifically apply this methodology to the hedging of barrier options, and highlight how the optimal hedging strategy undergoes distortions as the agent moves from being risk-averse to risk-seeking. As well as how the agent robustifies their strategy. We further investigate the performance of the hedge when the data generating process (DGP) varies from the training DGP, and demonstrate that the robust strategies outperform the non-robust ones., Comment: 18 pages, 14 figures, 1 table

We introduce a new approach for generating sequences of implied volatility (IV) surfaces across multiple assets that is faithful to historical prices. We do so using a combination of functional data analysis and neural stochastic differential equations (SDEs) combined with a probability integral transform penalty to reduce model misspecification. We demonstrate that learning the joint dynamics of IV surfaces and prices produces market scenarios that are consistent with historical features and lie within the sub-manifold of surfaces that are essentially free of static arbitrage. Finally, we demonstrate that delta hedging using the simulated surfaces generates profit and loss (P&L) distributions that are consistent with realised P&Ls., Comment: 38 pages, 19 figures, 5 tables

We introduce a distributional method for learning the optimal policy in risk averse Markov decision process with finite state action spaces, latent costs, and stationary dynamics. We assume sequential observations of states, actions, and costs and assess the performance of a policy using dynamic risk measures constructed from nested Kusuoka-type conditional risk mappings. For such performance criteria, randomized policies may outperform deterministic policies, therefore, the candidate policies lie in the d-dimensional simplex where d is the cardinality of the action space. Existing risk averse reinforcement learning methods seldom concern randomized policies, na\"ive extensions to current setting suffer from the curse of dimensionality. By exploiting certain structures embedded in the corresponding dynamic programming principle, we propose a distributional learning method for seeking the optimal policy. The conditional distribution of the value function is casted into a specific type of function, which is chosen with in mind the ease of risk averse optimization. We use a deep neural network to approximate said function, illustrate that the proposed method avoids the curse of dimensionality in the exploration phase, and explore the method's performance with a wide range of model parameters that are picked randomly.

In this paper, we use mean field games (MFGs) to investigate approximations of $N$-player games with uniformly symmetrically continuous heterogeneous closed-loop actions. To incorporate agents' risk aversion (beyond the classical expected utility of total costs), we use an abstract evaluation functional for their performance criteria. Centered around the notion of exploitability, we conduct non-asymptotic analysis on the approximation capability of MFGs from the perspective of state-action distributions without requiring the uniqueness of equilibria. Under suitable assumptions, we first show that scenarios in the $N$-player games with large $N$ and small average exploitabilities can be well approximated by approximate solutions of MFGs with relatively small exploitabilities. We then show that $\delta$-mean field equilibria can be used to construct $\varepsilon$-equilibria in $N$-player games. Furthermore, in this general setting, we prove the existence of mean field equilibria. This proof reveals a possible avenue for incorporating penalization for randomized action into MFGs.

We study optimal control in models with latent factors where the agent controls the distribution over actions, rather than actions themselves, in both discrete and continuous time. To encourage exploration of the state space, we reward exploration with Tsallis Entropy and derive the optimal distribution over states - which we prove is $q$-Gaussian distributed with location characterized through the solution of an FBS$\Delta$E and FBSDE in discrete and continuous time, respectively. We discuss the relation between the solutions of the optimal exploration problems and the standard dynamic optimal control solution. Finally, we develop the optimal policy in a model-agnostic setting along the lines of soft $Q$-learning. The approach may be applied in, e.g., developing more robust statistical arbitrage trading strategies.

Stress testing, and in particular, reverse stress testing, is a prominent exercise in risk management practice. Reverse stress testing, in contrast to (forward) stress testing, aims to find an alternative but plausible model such that under that alternative model, specific adverse stresses (i.e. constraints) are satisfied. Here, we propose a reverse stress testing framework for dynamic models. Specifically, we consider a compound Poisson process over a finite time horizon and stresses composed of expected values of functions applied to the process at the terminal time. We then define the stressed model as the probability measure under which the process satisfies the constraints and which minimizes the Kullback-Leibler divergence to the reference compound Poisson model. We solve this optimization problem, prove existence and uniqueness of the stressed probability measure, and provide a characterization of the Radon-Nikodym derivative from the reference model to the stressed model. We find that under the stressed measure, the intensity and the severity distribution of the process depend on time and state. We illustrate the dynamic stress testing by considering stresses on VaR and both VaR and CVaR jointly and provide illustrations of how the stochastic process is altered under these stresses. We generalize the framework to multivariate compound Poisson processes and stresses at times other than the terminal time. We illustrate the applicability of our framework by considering ``what if'' scenarios, where we answer the question: What is the severity of a stress on a portfolio component at an earlier time such that the aggregate portfolio exceeds a risk threshold at the terminal time? Furthermore, for general constraints, we propose an algorithm to simulate sample paths under the stressed measure, thus allowing to compare the effects of stresses on the dynamics of the process.

Given an n-dimensional stochastic process X driven by P-Brownian motions and Poisson random measures, we seek the probability measure Q, with minimal relative entropy to P, such that the Q-expectations of some terminal and running costs are constrained. We prove existence and uniqueness of the optimal probability measure, derive the explicit form of the measure change, and characterise the optimal drift and compensator adjustments under the optimal measure. We provide an analytical solution for Value-at-Risk (quantile) constraints, discuss how to perturb a Brownian motion to have arbitrary variance, and show that pinned measures arise as a limiting case of optimal measures. The results are illustrated in a risk management setting -- including an algorithm to simulate under the optimal measure -- where an agent seeks to answer the question: what dynamics are induced by a perturbation of the Value-at-Risk and the average time spent below a barrier on the reference process?

We propose a novel framework to solve risk-sensitive reinforcement learning (RL) problems where the agent optimises time-consistent dynamic spectral risk measures. Based on the notion of conditional elicitability, our methodology constructs (strictly consistent) scoring functions that are used as penalizers in the estimation procedure. Our contribution is threefold: we (i) devise an efficient approach to estimate a class of dynamic spectral risk measures with deep neural networks, (ii) prove that these dynamic spectral risk measures may be approximated to any arbitrary accuracy using deep neural networks, and (iii) develop a risk-sensitive actor-critic algorithm that uses full episodes and does not require any additional nested transitions. We compare our conceptually improved reinforcement learning algorithm with the nested simulation approach and illustrate its performance in two settings: statistical arbitrage and portfolio allocation on both simulated and real data., Comment: 41 pages, 7 figures

By adopting a distributional viewpoint on law-invariant convex risk measures, we construct dynamics risk measures (DRMs) at the distributional level. We then apply these DRMs to investigate Markov decision processes, incorporating latent costs, random actions, and weakly continuous transition kernels. Furthermore, the proposed DRMs allow risk aversion to change dynamically. Under mild assumptions, we derive a dynamic programming principle and show the existence of an optimal policy in both finite and infinite time horizons. Moreover, we provide a sufficient condition for the optimality of deterministic actions. For illustration, we conclude the paper with examples from optimal liquidation with limit order books and autonomous driving.

We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor-critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibility of our approach by applying it to three optimization problems: statistical arbitrage trading strategies, financial hedging, and obstacle avoidance robot control., Comment: 26 pages, 9 figures

Principal agent games are a growing area of research which focuses on the optimal behaviour of a principal and an agent, with the former contracting work from the latter, in return for providing a monetary award. While this field canonically considers a single agent, the situation where multiple agents, or even an infinite amount of agents are contracted by a principal are growing in prominence and pose interesting and realistic problems. Here, agents form a Nash equilibrium among themselves, and a Stackelberg equilibrium between themselves as a collective and the principal. We apply this framework to the problem of implementing Renewable Energy Certificate (REC) markets, where the principal requires regulated firms (power generators) to pay a non-compliance penalty which is inversely proportional to the amount of RECs they have. RECs can be obtained by generating electricity from clean sources or purchasing on the market. The agents react to this penalty and optimize their behaviours to navigate the system at minimum cost. In the agents' model we incorporate market clearing as well as agent heterogeneity. For a given market design, we find the Nash equilibrium among agents using techniques from mean field games. We then use techniques from extended McKean-Vlasov control problems to solve the principal (regulators) problem, who aim to choose the penalty function in such a way that balances environmental and revenue impacts optimally. We find through these techniques that the optimal penalty function is linear in the agents' state, suggesting the optimal emissions regulation market is more akin to a tax or rebate, regardless of the principal's utility function., Comment: Work in progress

Here, we develop a deep learning algorithm for solving Principal-Agent (PA) mean field games with market-clearing conditions -- a class of problems that have thus far not been studied and one that poses difficulties for standard numerical methods. We use an actor-critic approach to optimization, where the agents form a Nash equilibria according to the principal's penalty function, and the principal evaluates the resulting equilibria. The inner problem's Nash equilibria is obtained using a variant of the deep backward stochastic differential equation (BSDE) method modified for McKean-Vlasov forward-backward SDEs that includes dependence on the distribution over both the forward and backward processes. The outer problem's loss is further approximated by a neural net by sampling over the space of penalty functions. We apply our approach to a stylized PA problem arising in Renewable Energy Certificate (REC) markets, where agents may rent clean energy production capacity, trade RECs, and expand their long-term capacity to navigate the market at maximum profit. Our numerical results illustrate the efficacy of the algorithm and lead to interesting insights into the nature of optimal PA interactions in the mean-field limit of these markets., Comment: 22 pages, 8 Figures, 4 Tables

High-resolution spectroscopic surveys of the Milky Way have entered the Big Data regime and have opened avenues for solving outstanding questions in Galactic archaeology. However, exploiting their full potential is limited by complex systematics, whose characterization has not received much attention in modern spectroscopic analyses. In this work, we present a novel method to disentangle the component of spectral data space intrinsic to the stars from that due to systematics. Using functional principal component analysis on a sample of $18,933$ giant spectra from APOGEE, we find that the intrinsic structure above the level of observational uncertainties requires ${\approx}$10 functional principal components (FPCs). Our FPCs can reduce the dimensionality of spectra, remove systematics, and impute masked wavelengths, thereby enabling accurate studies of stellar populations. To demonstrate the applicability of our FPCs, we use them to infer stellar parameters and abundances of 28 giants in the open cluster M67. We employ Sequential Neural Likelihood, a simulation-based Bayesian inference method that learns likelihood functions using neural density estimators, to incorporate non-Gaussian effects in spectral likelihoods. By hierarchically combining the inferred abundances, we limit the spread of the following elements in M67: $\mathrm{Fe} \lesssim 0.02$ dex; $\mathrm{C} \lesssim 0.03$ dex; $\mathrm{O}, \mathrm{Mg}, \mathrm{Si}, \mathrm{Ni} \lesssim 0.04$ dex; $\mathrm{Ca} \lesssim 0.05$ dex; $\mathrm{N}, \mathrm{Al} \lesssim 0.07$ dex (at 68% confidence). Our constraints suggest a lack of self-pollution by core-collapse supernovae in M67, which has promising implications for the future of chemical tagging to understand the star formation history and dynamical evolution of the Milky Way., Comment: 30 pages, 16 figures, 5 tables, Accepted to ApJ, Code available at https://github.com/aaryapatil/specdims

We present a reinforcement learning (RL) approach for robust optimisation of risk-aware performance criteria. To allow agents to express a wide variety of risk-reward profiles, we assess the value of a policy using rank dependent expected utility (RDEU). RDEU allows the agent to seek gains, while simultaneously protecting themselves against downside risk. To robustify optimal policies against model uncertainty, we assess a policy not by its distribution, but rather, by the worst possible distribution that lies within a Wasserstein ball around it. Thus, our problem formulation may be viewed as an actor/agent choosing a policy (the outer problem), and the adversary then acting to worsen the performance of that strategy (the inner problem). We develop explicit policy gradient formulae for the inner and outer problems, and show its efficacy on three prototypical financial problems: robust portfolio allocation, optimising a benchmark, and statistical arbitrage., Comment: 12 pages, 5 figures

We propose a hybrid method for generating arbitrage-free implied volatility (IV) surfaces consistent with historical data by combining model-free Variational Autoencoders (VAEs) with continuous time stochastic differential equation (SDE) driven models. We focus on two classes of SDE models: regime switching models and L\'evy additive processes. By projecting historical surfaces onto the space of SDE model parameters, we obtain a distribution on the parameter subspace faithful to the data on which we then train a VAE. Arbitrage-free IV surfaces are then generated by sampling from the posterior distribution on the latent space, decoding to obtain SDE model parameters, and finally mapping those parameters to IV surfaces. We further refine the VAE model by including conditional features and demonstrate its superior generative out-of-sample performance., Comment: 20 pages, 7 figures

Trading frictions are stochastic. They are, moreover, in many instances fast-mean reverting. Here, we study how to optimally trade in a market with stochastic price impact and study approximations to the resulting optimal control problem using singular perturbation methods. We prove, by constructing sub- and super-solutions, that the approximations are accurate to the specified order. Finally, we perform some numerical experiments to illustrate the effect that stochastic trading frictions have on optimal trading., Comment: 21 pages, 6 figures, 3 tables

We study the problem of active portfolio management where an investor aims to outperform a benchmark strategy's risk profile while not deviating too far from it. Specifically, an investor considers alternative strategies whose terminal wealth lie within a Wasserstein ball surrounding a benchmark's -- being distributionally close -- and that have a specified dependence/copula -- tying state-by-state outcomes -- to it. The investor then chooses the alternative strategy that minimises a distortion risk measure of terminal wealth. In a general (complete) market model, we prove that an optimal dynamic strategy exists and provide its characterisation through the notion of isotonic projections. We further propose a simulation approach to calculate the optimal strategy's terminal wealth, making our approach applicable to a wide range of market models. Finally, we illustrate how investors with different copula and risk preferences invest and improve upon the benchmark using the Tail Value-at-Risk, inverse S-shaped, and lower- and upper-tail distortion risk measures as examples. We find that investors' optimal terminal wealth distribution has larger probability masses in regions that reduce their risk measure relative to the benchmark while preserving the benchmark's structure., Comment: 37 pages, 2 tables, 6 figures

We study a general class of entropy-regularized multi-variate LQG mean field games (MFGs) in continuous time with $K$ distinct sub-population of agents. We extend the notion of actions to action distributions (exploratory actions), and explicitly derive the optimal action distributions for individual agents in the limiting MFG. We demonstrate that the optimal set of action distributions yields an $\epsilon$-Nash equilibrium for the finite-population entropy-regularized MFG. Furthermore, we compare the resulting solutions with those of classical LQG MFGs and establish the equivalence of their existence., Comment: To appear in Automatica

We develop the optimal trading strategy for a foreign exchange (FX) broker who must liquidate a large position in an illiquid currency pair. To maximize revenues, the broker considers trading in a currency triplet which consists of the illiquid pair and two other liquid currency pairs. The liquid pairs in the triplet are chosen so that one of the pairs is redundant. The broker is risk-neutral and accounts for model ambiguity in the FX rates to make her strategy robust to model misspecification. When the broker is ambiguity neutral (averse) the trading strategy in each pair is independent (dependent) of the inventory in the other two pairs in the triplet. We employ simulations to illustrate how the robust strategies perform. For a range of ambiguity aversion parameters, we find the mean Profit and Loss (P&L) of the strategy increases and the standard deviation of the P&L decreases as ambiguity aversion increases., Comment: 35 pages, 14 figures, 1 table

We address the Merton problem of maximizing the expected utility of terminal wealth using techniques from variational analysis. Under a general continuous semimartingale market model with stochastic parameters, we obtain a characterization of the optimal portfolio for general utility functions in terms of a forward-backward stochastic differential equation (FBSDE) and derive solutions for a number of well-known utility functions. Our results complement a previous studies conducted on optimal strategies in markets driven by Brownian noise with random drift and volatility parameters.

Solar Renewable Energy Certificate (SREC) markets are a market-based system that incentivizes solar energy generation. A regulatory body imposes a lower bound on the amount of energy each regulated firm must generate via solar means, providing them with a tradeable certificate for each MWh generated. Firms seek to navigate the market optimally by modulating their SREC generation and trading rates. As such, the SREC market can be viewed as a stochastic game, where agents interact through the SREC price. We study this stochastic game by solving the mean-field game (MFG) limit with sub-populations of heterogeneous agents. Market participants optimize costs accounting for trading frictions, cost of generation, non-linear non-compliance costs, and generation uncertainty. Moreover, we endogenize SREC price through market clearing. We characterize firms' optimal controls as the solution of McKean-Vlasov (MV) FBSDEs and determine the equilibrium SREC price. We establish the existence and uniqueness of a solution to this MV-FBSDE, and prove that the MFG strategies form an $\epsilon$-Nash equilibrium for the finite player game. Finally, we develop a numerical scheme for solving the MV-FBSDEs and conduct a simulation study., Comment: 40 pages, 8 figures

Latency (i.e., time delay) in electronic markets affects the efficacy of liquidity taking strategies. During the time liquidity takers process information and send marketable limit orders (MLOs) to the exchange, the limit order book (LOB) might undergo updates, so there is no guarantee that MLOs are filled. We develop a latency-optimal trading strategy that improves the marksmanship of liquidity takers. The interaction between the LOB and MLOs is modelled as a marked point process. Each MLO specifies a price limit so the order can receive worse prices and quantities than those the liquidity taker targets if the updates in the LOB are against the interest of the trader. In our model, the liquidity taker balances the tradeoff between missing trades and the costs of walking the book. We employ techniques of variational analysis to obtain the optimal price limit of each MLO the agent sends. The price limit of a MLO is characterized as the solution to a new class of forward-backward stochastic differential equations (FBSDEs) driven by random measures. We prove the existence and uniqueness of the solution to the FBSDE and numerically solve it to illustrate the performance of the latency-optimal strategies.

A risk-averse agent hedges her exposure to a non-tradable risk factor $U$ using a correlated traded asset $S$ and accounts for the impact of her trades on both factors. The effect of the agent's trades on $U$ is referred to as cross-impact. By solving the agent's stochastic control problem, we obtain a closed-form expression for the optimal strategy when the agent holds a linear position in $U$. When the exposure to the non-tradable risk factor $\psi(U_T)$ is non-linear, we provide an approximation to the optimal strategy in closed-form, and prove that the value function is correctly approximated by this strategy when cross-impact and risk-aversion are small. We further prove that when $\psi(U_T)$ is non-linear, the approximate optimal strategy can be written in terms of the optimal strategy for a linear exposure with the size of the position changing dynamically according to the exposure's "Delta" under a particular probability measure., Comment: Originally posted to SSRN April 27, 2018. Forthcoming in Mathematical Finance

We present an overview of the broad class of financial models in which the prices of assets are L\'evy-Ito processes driven by an $n$-dimensional Brownian motion and an independent Poisson random measure. The Poisson random measure is associated with an $n$-dimensional L\'evy process. Each model consists of a pricing kernel, a money market account, and one or more risky assets. We show how the excess rate of return above the interest rate can be calculated for risky assets in such models, thus showing the relationship between risk and return when asset prices have jumps. The framework is applied to a variety of asset classes, allowing one to construct new models as well as interesting generalizations of familiar models., Comment: 42 pages

Model-free learning for multi-agent stochastic games is an active area of research. Existing reinforcement learning algorithms, however, are often restricted to zero-sum games, and are applicable only in small state-action spaces or other simplified settings. Here, we develop a new data efficient Deep-Q-learning methodology for model-free learning of Nash equilibria for general-sum stochastic games. The algorithm uses a local linear-quadratic expansion of the stochastic game, which leads to analytically solvable optimal actions. The expansion is parametrized by deep neural networks to give it sufficient flexibility to learn the environment without the need to experience all state-action pairs. We study symmetry properties of the algorithm stemming from label-invariant stochastic games and as a proof of concept, apply our algorithm to learning optimal trading strategies in competitive electronic markets., Comment: 15 pages, 3 figures

SREC markets are a relatively novel market-based system to incentivize the production of energy from solar means. A regulator imposes a floor on the amount of energy each regulated firm must generate from solar power in a given period and provides them with certificates for each generated MWh. Firms offset these certificates against the floor and pay a penalty for any lacking certificates. Certificates are tradable assets, allowing firms to purchase/sell them freely. In this work, we formulate a stochastic control problem for generating and trading in SREC markets from a regulated firm's perspective. We account for generation and trading costs, the impact both have on SREC prices, provide a characterization of the optimal strategy, and develop a numerical algorithm to solve this control problem. Through numerical experiments, we explore how a firm who acts optimally behaves under various conditions. We find that an optimal firm's generation and trading behaviour can be separated into various regimes, based on the marginal benefit of obtaining an additional SREC, and validate our theoretical characterization of the optimal strategy. We also conduct parameter sensitivity experiments and conduct comparisons of the optimal strategy to other candidate strategies., Comment: 29 pages, 12 figures

We address a portfolio selection problem that combines active (outperformance) and passive (tracking) objectives using techniques from convex analysis. We assume a general semimartingale market model where the assets' growth rate processes are driven by a latent factor. Using techniques from convex analysis we obtain a closed-form solution for the optimal portfolio and provide a theorem establishing its uniqueness. The motivation for incorporating latent factors is to achieve improved growth rate estimation, an otherwise notoriously difficult task. To this end, we focus on a model where growth rates are driven by an unobservable Markov chain. The solution in this case requires a filtering step to obtain posterior probabilities for the state of the Markov chain from asset price information, which are subsequently used to find the optimal allocation. We show the optimal strategy is the posterior average of the optimal strategies the investor would have held in each state assuming the Markov chain remains in that state. Finally, we implement a number of historical backtests to demonstrate the performance of the optimal portfolio.

Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios., Comment: 20 pages, 7 figures, 1 table. Updated minor typos

We develop a convex analysis approach for solving LQG optimal control problems and apply it to major-minor (MM) LQG mean-field game (MFG) systems. The approach retrieves the best response strategies for the major agent and all minor agents that attain an $\epsilon$-Nash equilibrium. An important and distinctive advantage to this approach is that unlike the classical approach in the literature, we are able to avoid imposing assumptions on the evolution of the mean-field. In particular, this provides a tool for dealing with complex and non-standard systems., Comment: To appear in Systems & Control Letters

Even when confronted with the same data, agents often disagree on a model of the real-world. Here, we address the question of how interacting heterogenous agents, who disagree on what model the real-world follows, optimize their trading actions. The market has latent factors that drive prices, and agents account for the permanent impact they have on prices. This leads to a large stochastic game, where each agents' performance criteria are computed under a different probability measure. We analyse the mean-field game (MFG) limit of the stochastic game and show that the Nash equilibrium is given by the solution to a non-standard vector-valued forward-backward stochastic differential equation. Under some mild assumptions, we construct the solution in terms of expectations of the filtered states. Furthermore, we prove the MFG strategy forms an $\epsilon$-Nash equilibrium for the finite player game. Lastly, we present a least-squares Monte Carlo based algorithm for computing the equilibria and show through simulations that increasing disagreement may increase price volatility and trading activity., Comment: 36 pages, 3 figures

In many stochastic games stemming from financial models, the environment evolves with latent factors and there may be common noise across agents' states. Two classic examples are: (i) multi-agent trading on electronic exchanges, and (ii) systemic risk induced through inter-bank lending/borrowing. Moreover, agents' actions often affect the environment, and some agent's may be small while others large. Hence sub-population of agents may act as minor agents, while another class may act as major agents. To capture the essence of such problems, here, we introduce a general class of non-cooperative heterogeneous stochastic games with one major agent and a large population of minor agents where agents interact with an observed common process impacted by the mean field. A latent Markov chain and a latent Wiener process (common noise) modulate the common process, and agents cannot observe them. We use filtering techniques coupled with a convex analysis approach to (i) solve the mean field game limit of the problem, (ii) demonstrate that the best response strategies generate an $\epsilon$-Nash equilibrium for finite populations, and (iii) obtain explicit characterisations of the best response strategies., Comment: 28 pages

Executing a basket of co-integrated assets is an important task facing investors. Here, we show how to do this accounting for the informational advantage gained from assets within and outside the basket, as well as for the permanent price impact of market orders (MOs) from all market participants, and the temporary impact that the agent's MOs have on prices. The execution problem is posed as an optimal stochastic control problem and we demonstrate that, under some mild conditions, the value function admits a closed-form solution, and prove a verification theorem. Furthermore, we use data of five stocks traded in the Nasdaq exchange to estimate the model parameters and use simulations to illustrate the performance of the strategy. As an example, the agent liquidates a portfolio consisting of shares in Intel Corporation (INTC) and Market Vectors Semiconductor ETF (SMH). We show that including the information provided by three additional assets, FARO Technologies (FARO), NetApp (NTAP) and Oracle Corporation (ORCL), considerably improves the strategy's performance; for the portfolio we execute, it outperforms the multi-asset version of Almgren-Chriss by approximately 4 to 4.5 basis points., Comment: 32 pages,4 figures, 11 tables. First appeared on SSRN Oct 2015 at http://ssrn.com/abstract=2681309

Alpha signals for statistical arbitrage strategies are often driven by latent factors. This paper analyses how to optimally trade with latent factors that cause prices to jump and diffuse. Moreover, we account for the effect of the trader's actions on quoted prices and the prices they receive from trading. Under fairly general assumptions, we demonstrate how the trader can learn the posterior distribution over the latent states, and explicitly solve the latent optimal trading problem. We provide a verification theorem, and a methodology for calibrating the model by deriving a variation of the expectation-maximization algorithm. To illustrate the efficacy of the optimal strategy, we demonstrate its performance through simulations and compare it to strategies which ignore learning in the latent factors. We also provide calibration results for a particular model using Intel Corporation stock as an example., Comment: 42 pages, 5 figures

We examine the Foreign Exchange (FX) spot price spreads with and without Last Look on the transaction. We assume that brokers are risk-neutral and they quote spreads so that losses to latency arbitrageurs (LAs) are recovered from other traders in the FX market. These losses are reduced if the broker can reject, ex-post, loss-making trades by enforcing the Last Look option which is a feature of some trading venues in FX markets. For a given rejection threshold the risk-neutral broker quotes a spread to the market so that her expected profits are zero. When there is only one venue, we find that the Last Look option reduces quoted spreads. If there are two venues we show that the market reaches an equilibrium where traders have no incentive to migrate. The equilibrium can be reached with both venues coexisting, or with only one venue surviving. Moreover, when one venue enforces Last Look and the other one does not, counterintuitively, it may be the case that the Last Look venue quotes larger spreads., Comment: 40 pages, 7 figures

We develop a mixed least squares Monte Carlo-partial differential equation (LSMC-PDE) method for pricing Bermudan style options on assets whose volatility is stochastic. The algorithm is formulated for an arbitrary number of assets and volatility processes and we prove the algorithm converges almost surely for a class of models. We also discuss two methods to improve the algorithm's computational complexity. Our numerical examples focus on the single ($2d$) and multi-dimensional ($4d$) Heston models and we compare our hybrid algorithm with classical LSMC approaches. In each case, we find that the hybrid algorithm outperforms standard LSMC in terms of estimating prices and optimal exercise boundaries., Comment: The first version of this paper was submitted to SSRN in November 2016: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2870962

We introduce a new approach to incorporate uncertainty into the decision to invest in a commodity reserve. The investment is an irreversible one-off capital expenditure, after which the investor receives a stream of cashflow from extracting the commodity and selling it on the spot market. The investor is exposed to price uncertainty and uncertainty in the amount of available resources in the reserves (i.e. technical uncertainty). She does, however, learn about the reserve levels through time, which is a key determinant in the decision to invest. To model the reserve level uncertainty and how she learns about the estimates of the commodity in the reserve, we adopt a continuous-time Markov chain model to value the option to invest in the reserve and investigate the value that learning has prior to investment., Comment: Originally posted Oct 6 2014 on SSRN at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2505444

Portfolio management problems are often divided into two types: active and passive, where the objective is to outperform and track a preselected benchmark, respectively. Here, we formulate and solve a dynamic asset allocation problem that combines these two objectives in a unified framework. We look to maximize the expected growth rate differential between the wealth of the investor's portfolio and that of a performance benchmark while penalizing risk-weighted deviations from a given tracking portfolio. Using stochastic control techniques, we provide explicit closed-form expressions for the optimal allocation and we show how the optimal strategy can be related to the growth optimal portfolio. The admissible benchmarks encompass the class of functionally generated portfolios (FGPs), which include the market portfolio, as the only requirement is that they depend only on the prevailing asset values. Finally, some numerical experiments are presented to illustrate the risk-reward profile of the optimal allocation., Comment: Originally posted Jan 31 2017 on SSRN at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2908552

Financial markets are often driven by latent factors which traders cannot observe. Here, we address an algorithmic trading problem with collections of heterogeneous agents who aim to perform optimal execution or statistical arbitrage, where all agents filter the latent states of the world, and their trading actions have permanent and temporary price impact. This leads to a large stochastic game with heterogeneous agents. We solve the stochastic game by investigating its mean-field game (MFG) limit, with sub-populations of heterogeneous agents, and, using a convex analysis approach, we show that the solution is characterized by a vector-valued forward-backward stochastic differential equation (FBSDE). We demonstrate that the FBSDE admits a unique solution, obtain it in closed-form, and characterize the optimal behaviour of the agents in the MFG equilibrium. Moreover, we prove the MFG equilibrium provides an $\epsilon$-Nash equilibrium for the finite player game. We conclude by illustrating the behaviour of agents using the optimal MFG strategy through simulated examples., Comment: 34 pages, 1 figure