Search

Optimal decision-making under partial observability requires reasoning about the uncertainty of the environment's hidden state. However, most reinforcement learning architectures handle partial observability with sequence models that have no internal mechanism to incorporate uncertainty in their hidden state representation, such as recurrent neural networks, deterministic state-space models and transformers. Inspired by advances in probabilistic world models for reinforcement learning, we propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models and train it end-to-end within a model-free architecture to maximize returns. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan, which scales logarithmically with the sequence length. By design, Kalman filter layers are a drop-in replacement for other recurrent layers in standard model-free architectures, but importantly they include an explicit mechanism for probabilistic filtering of the latent state representation. Experiments in a wide variety of tasks with partial observability show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.

We consider a sequential decision making task, where the goal is to optimize an unknown function without evaluating parameters that violate an a~priori unknown (safety) constraint. A common approach is to place a Gaussian process prior on the unknown functions and allow evaluations only in regions that are safe with high probability. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. Moreover, the way in which they exploit regularity assumptions about the constraint introduces an additional critical hyperparameter. In this paper, we propose an information-theoretic safe exploration criterion that directly exploits the GP posterior to identify the most informative safe parameters to evaluate. The combination of this exploration criterion with a well known Bayesian optimization acquisition function yields a novel safe Bayesian optimization selection criterion. Our approach is naturally applicable to continuous domains and does not require additional explicit hyperparameters. We theoretically analyze the method and show that we do not violate the safety constraint with high probability and that we learn about the value of the safe optimum up to arbitrary precision. Empirical evaluations demonstrate improved data-efficiency and scalability., Comment: arXiv admin note: text overlap with arXiv:2212.04914

We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over Markov decision processes (MDPs). Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation (UBE), but the over-approximation may result in inefficient exploration. We propose a new UBE whose solution converges to the true posterior variance over values and leads to lower regret in tabular exploration problems. We identify challenges to apply the UBE theory beyond tabular problems and propose a suitable approximation. Based on this approximation, we introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC), that can be applied for either risk-seeking or risk-averse policy optimization with minimal changes. Experiments in both online and offline RL demonstrate improved performance compared to other uncertainty estimation methods., Comment: arXiv admin note: substantial text overlap with arXiv:2302.12526

Quantifying uncertainty about a policy's long-term performance is important to solve sequential decision-making tasks. We study the problem from a model-based Bayesian reinforcement learning perspective, where the goal is to learn the posterior distribution over value functions induced by parameter (epistemic) uncertainty of the Markov decision process. Previous work restricts the analysis to a few moments of the distribution over values or imposes a particular distribution shape, e.g., Gaussians. Inspired by distributional reinforcement learning, we introduce a Bellman operator whose fixed-point is the value distribution function. Based on our theory, we propose Epistemic Quantile-Regression (EQR), a model-based algorithm that learns a value distribution function. We combine EQR with soft actor-critic (SAC) for policy optimization with an arbitrary differentiable objective function of the learned value distribution. Evaluation across several continuous-control tasks shows performance benefits with respect to both model-based and model-free algorithms. The code is available at https://github.com/boschresearch/dist-mbrl.

We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation, but the over-approximation may result in inefficient exploration. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values and explicitly characterizes the gap in previous work. Moreover, our uncertainty quantification technique is easily integrated into common exploration strategies and scales naturally beyond the tabular setting by using standard deep reinforcement learning architectures. Experiments in difficult exploration tasks, both in tabular and continuous control settings, show that our sharper uncertainty estimates improve sample-efficiency., Comment: AISTATS 2023

We consider a sequential decision making task where we are not allowed to evaluate parameters that violate an a priori unknown (safety) constraint. A common approach is to place a Gaussian process prior on the unknown constraint and allow evaluations only in regions that are safe with high probability. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. Moreover, the way in which they exploit regularity assumptions about the constraint introduces an additional critical hyperparameter. In this paper, we propose an information-theoretic safe exploration criterion that directly exploits the GP posterior to identify the most informative safe parameters to evaluate. Our approach is naturally applicable to continuous domains and does not require additional hyperparameters. We theoretically analyze the method and show that we do not violate the safety constraint with high probability and that we explore by learning about the constraint up to arbitrary precision. Empirical evaluations demonstrate improved data-efficiency and scalability., Comment: Submitted to NeurIPS 2022

Physics-informed neural networks (PINNs) are an increasingly powerful way to solve partial differential equations, generate digital twins, and create neural surrogates of physical models. In this manuscript we detail the inner workings of NeuralPDE.jl and show how a formulation structured around numerical quadrature gives rise to new loss functions which allow for adaptivity towards bounded error tolerances. We describe the various ways one can use the tool, detailing mathematical techniques like using extended loss functions for parameter estimation and operator discovery, to help potential users adopt these PINN-based techniques into their workflow. We showcase how NeuralPDE uses a purely symbolic formulation so that all of the underlying training code is generated from an abstract formulation, and show how to make use of GPUs and solve systems of PDEs. Afterwards we give a detailed performance analysis which showcases the trade-off between training techniques on a large set of PDEs. We end by focusing on a complex multiphysics example, the Doyle-Fuller-Newman (DFN) Model, and showcase how this PDE can be formulated and solved with NeuralPDE. Together this manuscript is meant to be a detailed and approachable technical report to help potential users of the technique quickly get a sense of the real-world performance trade-offs and use cases of the PINN techniques., Comment: 74 pages, 20+ figures, 20+ tables

The aim of this work is to evaluate the feasibility of re-implementing some key parts of the widely used Weather Research and Forecasting WRF-SFIRE simulator by replacing its core differential equations numerical solvers with state-of-the-art physics-informed machine learning techniques to solve ODEs and PDEs, in order to transform it into a real-time simulator for wildfire spread prediction. The main programming language used is Julia, a compiled language which offers better perfomance than interpreted ones, providing Just in Time (JIT) compilation with different optimization levels. Moreover, Julia is particularly well suited for numerical computation and for the solution of complex physical models, both considering the syntax and the presence of some specific libraries such as DifferentialEquations.jl and ModellingToolkit.jl., Comment: Research presented at the competition "ProjectX 2020" by UofT AI (15 pages)

The results of recent photometric observations of suspected variable stars VV Cam and VW Cam and of fainter stars in their field of view are reported and compared to data available from scientific literature and public astronomical databases., Comment: 16 Pages, 10 Figures, 17 Tables. Appendix 1: BVRI photometric data of VV Cam. Appendix 2: BVRI photometric data of VW Cam

The numerical simulation of acoustic waves in complex 3D media is a key topic in many branches of science, from exploration geophysics to non-destructive testing and medical imaging. With the drastic increase in computing capabilities this field has dramatically grown in the last twenty years. However many 3D computations, especially at high frequency and/or long range, are still far beyond current reach and force researchers to resort to approximations, for example by working in 2D (plane strain) or by using a paraxial approximation. This article presents and validates a numerical technique based on an axisymmetric formulation of a spectral finite-element method in the time domain for heterogeneous fluid-solid media. Taking advantage of axisymmetry enables the study of relevant 3D configurations at a very moderate computational cost. The axisymmetric spectral-element formulation is first introduced, and validation tests are then performed. A typical application of interest in ocean acoustics showing upslope propagation above a dipping viscoelastic ocean bottom is then presented. The method correctly models backscattered waves and explains the transmission losses discrepancies pointed out in Jensen et al. (2007). Finally, a realistic application to a double seamount problem is considered., Comment: Added a reference, and fixed a typo (cylindrical versus spherical)

Complex cementitious matrices undergo weathering with environmental exchange and can release metallic pollutants during alteration. The molecular mechanisms responsible for metal release are difficult to identify, though this is necessary if such processes are to be controlled. The present study determines and models the molecular mechanisms of Pb release during Portland cement leaching. As Pb release is strongly related to its speciation (i.e. atomic environment and the nature of bearing phases), the first objective of the present study was to investigate the evolution of Pb retention sites together with the evolution of the cement mineralogy during leaching. Complementary and efficient investigation tools were used, namely X-ray diffraction, micro-X-ray fluorescence and X-ray absorption fine structures. The second objective was to reproduce our results with a reactive transport code (CHESS/HYTEC) in order to test the proposed speciation model of Pb. Combined results indicate that in both the unaltered core and the altered layer of the leached cement, Pb(II) would be retained through C-S-H 'nano-structure', probably linked to a Q1 or Q2P silicate tetrahedra. Moreover in the altered layer, the presence of Fe atoms in the atomic environment of Pb is highly probable. Unfortunately little is known about Fe phases in cement, which makes the interpretation difficult. Can Fe- substituted hydrogranet (C3AH6) be responsible for Pb retention? Modelling results were consistent with Pb retention through C-S-H in layers and also in an additional, possibly Fe-containing, Pb-retention phase in the altered layer.