Search

Functional data analysis is the study of statistical algorithms which are applied in the scenario when the observed data is a collection of functions. Since this type of data is becoming cheaper and easier to collect, there is an increased need to develop statistical tools to handle such data. The first part of this thesis focuses on deriving distances between distributions over function spaces and applying these to two-sample testing, goodness-of-fit testing and sample quality assessment. This presents a wide range of contributions since currently there exists either very few or no methods at all to tackle these problems for functional data. The second part of this thesis adopts the functional-analytic perspective to two statistical algorithms. This is a perspective where functions are viewed as living in specific function spaces and the tool box of functional analysis is applied to identify and prove properties of the algorithms. The two algorithms are variational Gaussian processes, used widely throughout machine learning for function modelling with large observation data sets, and functional statistical depth, used widely as a means to evaluate outliers and perform testing for functional data sets. The results presented contribute a taxonomy of the variational Gaussian process methodology and multiple new results in the theory of functional depth including the open problem of providing a depth which characterises distributions on function spaces.

Over the past decades, we have seen an increase in the availability of data that includes spatial information. Incorporating spatial information in models may result in performance improvements, which may then be used to better inform decision-making processes. When modelling spatial data, typical assumptions such as independence of observations across locations, no longer hold. As a consequence, careful methodology is required. This thesis addresses the modelling of two common types of data encountered in spatial modelling: measurements of a quantity at pre-specified locations (e.g., sensor measurements), and events for which geographical location and time are recorded. We develop effective approaches for modelling spatial data in an interpretable manner, thus making it suitable for application domains where the transparency of a model is a desired property. We demonstrate the developed approaches with empirical simulation studies.

In this thesis we build a geometric theory of Hamiltonian Monte Carlo, with an emphasis on symmetries and its bracket generalisations, construct the canonical geometry of smooth measures and Stein operators, and derive the complete recipe of measure-constraints preserving dynamics and diffusions on arbitrary manifolds. Specifically, we will explain the central role played by mechanics with symmetries to obtain efficient numerical integrators, and provide a general method to construct explicit integrators for HMC on geodesic orbit manifolds via symplectic reduction. Following ideas developed by Maxwell, Volterra, Poincaré, de Rham, Koszul, Dufour, Weinstein, and others, we will then show that any smooth distribution generates considerable geometric content, including ''musical" isomorphisms between multi-vector fields and twisted differential forms, and a boundary operator - the rotationnel, which, in particular, engenders the canonical Stein operator. We then introduce the ''bracket formalism" and its induced mechanics, a generalisation of Poisson mechanics and gradient flows that provides a general mechanism to associate unnormalised probability densities to flows depending on the score pointwise. Most importantly, we will characterise all measure-constraints preserving flows on arbitrary manifolds, showing the intimate relation between measure-preserving Nambu mechanics and closed twisted forms. Our results are canonical. As a special case we obtain the characterisation of measure-preserving bracket mechanical systems and measure-preserving diffusions, thus explaining and extending to manifolds the complete recipe of SGMCMC. We will discuss the geometry of Stein operators and extend the density approach by showing these are simply a reformulation of the exterior derivative on twisted forms satisfying Stokes' theorem. Combining the canonical Stein operator with brackets allows us to naturally recover the Riemannian and diffusion Stein operators as special cases. Finally, we shall introduce the minimum Stein discrepancy estimators, which provide a unifying perspective of parameter inference based on score matching, contrastive divergence, and minimum probability flow.

Modelling, simulation and experimentation are each important to understand the world that we live in. The world itself is a complex system, although the physical and socio-economic systems within are usually studied separately with a particular objective in mind. Owing to economic, practical and ethical considerations, experimentation and direct measurement are not always possible for these systems. Instead, a mathematical model can be used to retrieve the unknown quantities of interest from indirect measurement data, and to study how the system will behave in a number of different scenarios. Both tasks require solving the so-called parameter estimation problem. There are a number of uncertainties encountered in the modelling process, including those introduced by the mathematical model and its numerical simulation, and those present in the measurement data. A statistical approach to parameter estimation allows these uncertainties to be formally accounted for. There are a number advantages for using the Bayesian framework, although there are usually non-trivial statistical and computational challenges that must first be overcome. In this thesis, some new Bayesian approaches are developed to provide improved modelling capabilities for physical and socio-economic systems. The systems studied in this thesis involve spatially-distributed data, which includes the estimation of a spatially varying parameter via a complex function, parameter estimation for high-dimensional Gaussian models and uncertainty quantification for urban simulations. Monte Carlo methods are used throughout to obtain accurate summaries of the complex probability distributions involved. The new approaches are demonstrated with empirical simulation studies. Whilst the new approaches are shown to provide improvements on the existing ones, Bayesian inference remains a challenging and computationally intensive task. Further work is suggested to accelerate Bayesian inference, so that inferences can be made on more practical time scales.

Imagine a world where computational simulations can be inverted as easily as running them forwards, where data can be used to refine models automatically, and where the only expertise one needs to carry out powerful statistical analysis is a basic proficiency in scientific coding. Creating such a world is the ambitious long-term aim of probabilistic programming. The bottleneck for improving the probabilistic models, or simulators, used throughout the quantitative sciences, is often not an ability to devise better models conceptually, but a lack of expertise, time, or resources to realize such innovations. Probabilistic programming systems (PPSs) help alleviate this bottleneck by providing an expressive and accessible modeling framework, then automating the required computation to draw inferences from the model, for example finding the model parameters likely to give rise to a certain output. By decoupling model specification and inference, PPSs streamline the process of developing and drawing inferences from new models, while opening up powerful statistical methods to non-experts. Many systems further provide the flexibility to write new and exciting models which would be hard, or even impossible, to convey using conventional statistical frameworks. The central goal of this thesis is to improve and extend PPSs. In particular, we will make advancements to the underlying inference engines and increase the range of problems which can be tackled. For example, we will extend PPSs to a mixed inference-optimization framework, thereby providing automation of tasks such as model learning and engineering design. Meanwhile, we make inroads into constructing systems for automating adaptive sequential design problems, providing potential applications across the sciences. Furthermore, the contributions of the work reach far beyond probabilistic programming, as achieving our goal will require us to make advancements in a number of related fields such as particle Markov chain Monte Carlo methods, Bayesian optimization, and Monte Carlo fundamentals.