Search

In this work, we develop a new computational framework to solve sequential Bayesian experimental design (SBOED) problems constrained by large-scale partial differential equations with infinite-dimensional random parameters. We propose an adaptive terminal formulation of the optimality criteria for SBOED to achieve adaptive global optimality. We also establish an equivalent optimization formulation to achieve computational simplicity enabled by Laplace and low-rank approximations of the posterior. To accelerate the solution of the SBOED problem, we develop a derivative-informed latent attention neural operator (LANO), a new neural network surrogate model that leverages (1) derivative-informed dimension reduction for latent encoding, (2) an attention mechanism to capture the dynamics in the latent space, (3) an efficient training in the latent space augmented by projected Jacobian, which collectively lead to an efficient, accurate, and scalable surrogate in computing not only the parameter-to-observable (PtO) maps but also their Jacobians. We further develop the formulation for the computation of the MAP points, the eigenpairs, and the sampling from posterior by LANO in the reduced spaces and use these computations to solve the SBOED problem. We demonstrate the superior accuracy of LANO compared to two other neural architectures and the high accuracy of LANO compared to the finite element method (FEM) for the computation of MAP points in solving the SBOED problem with application to the experimental design of the time to take MRI images in monitoring tumor growth.

This paper introduces the Inside-Out Nested Particle Filter (IO-NPF), a novel, fully recursive, algorithm for amortized sequential Bayesian experimental design in the non-exchangeable setting. We frame policy optimization as maximum likelihood estimation in a non-Markovian state-space model, achieving (at most) $\mathcal{O}(T^2)$ computational complexity in the number of experiments. We provide theoretical convergence guarantees and introduce a backward sampling algorithm to reduce trajectory degeneracy. IO-NPF offers a practical, extensible, and provably consistent approach to sequential Bayesian experimental design, demonstrating improved efficiency over existing methods.

Conventional Bayesian optimal experimental design seeks to maximize the expected information gain (EIG) on model parameters. However, the end goal of the experiment often is not to learn the model parameters, but to predict downstream quantities of interest (QoIs) that depend on the learned parameters. And designs that offer high EIG for parameters may not translate to high EIG for QoIs. Goal-oriented optimal experimental design (GO-OED) thus directly targets to maximize the EIG of QoIs. We introduce LF-GO-OED (likelihood-free goal-oriented optimal experimental design), a computational method for conducting GO-OED with nonlinear observation and prediction models. LF-GO-OED is specifically designed to accommodate implicit models, where the likelihood is intractable. In particular, it builds a density ratio estimator from samples generated from approximate Bayesian computation (ABC), thereby sidestepping the need for likelihood evaluations or density estimations. The overall method is validated on benchmark problems with existing methods, and demonstrated on scientific applications of epidemiology and neural science.

The study of experimental design offers tremendous benefits for answering causal questions across a wide range of applications, including agricultural experiments, clinical trials, industrial experiments, social experiments, and digital experiments. Although valuable in such applications, the costs of experiments often drive experimenters to seek more efficient designs. Recently, experimenters have started to examine such efficiency questions from an optimization perspective, as experimental design problems are fundamentally decision-making problems. This perspective offers a lot of flexibility in leveraging various existing optimization tools to study experimental design problems. This manuscript thus aims to examine the foundations of experimental design problems in the context of causal inference as viewed through an optimization lens.

This paper develops a Bayesian optimal experimental design for robot kinematic calibration on ${\mathbb{S}^3 \!\times\! \mathbb{R}^3}$. Our method builds upon a Gaussian process approach that incorporates a geometry-aware kernel based on Riemannian Mat\'ern kernels over ${\mathbb{S}^3}$. To learn the forward kinematics errors via Bayesian optimization with a Gaussian process, we define a geodesic distance-based objective function. Pointwise values of this function are sampled via noisy measurements taken through fiducial markers on the end-effector using a camera and computed pose with the nominal kinematics. The corrected Denavit-Hartenberg parameters are obtained using an efficient quadratic program that operates on the collected data sets. The effectiveness of the proposed method is demonstrated via simulations and calibration experiments on NASA's ocean world lander autonomy testbed (OWLAT).

Optimal experimental design (OED) concerns itself with identifying ideal methods of data collection, e.g.~via sensor placement. The \emph{greedy algorithm}, that is, placing one sensor at a time, in an iteratively optimal manner, stands as an extremely robust and easily executed algorithm for this purpose. However, it is a priori unclear whether this algorithm leads to sub-optimal regimes. Taking advantage of the author's recent work on non-smooth convex optimality criteria for OED, we here present a framework for verifying global optimality for the greedy algorithm, as well as employing gradient-based speed-ups.

Randomized experiments are considered the gold standard for estimating causal effects. However, out of the set of possible randomized assignments, some may be likely to produce poor effect estimates and misleading conclusions. Restricted randomization is an experimental design strategy that filters out undesirable treatment assignments, but its application has primarily been limited to ensuring covariate balance in two-arm studies where the target estimand is the average treatment effect. Other experimental settings with different design desiderata and target effect estimands could also stand to benefit from a restricted randomization approach. We introduce Inspection-Guided Randomization (IGR), a transparent and flexible framework for restricted randomization that filters out undesirable treatment assignments by inspecting assignments against analyst-specified, domain-informed design desiderata. In IGR, the acceptable treatment assignments are locked in ex ante and pre-registered in the trial protocol, thus safeguarding against $p$-hacking and promoting reproducibility. Through illustrative simulation studies motivated by education and behavioral health interventions, we demonstrate how IGR can be used to improve effect estimates compared to benchmark designs in group formation experiments and experiments with interference.

Complex dynamic systems are typically either modeled using expert knowledge in the form of differential equations or via data-driven universal approximation models such as artificial neural networks (ANN). While the first approach has advantages with respect to interpretability, transparency, data-efficiency, and extrapolation, the second approach is able to learn completely unknown functional relations from data and may result in models that can be evaluated more efficiently. To combine the complementary advantages, universal differential equations (UDE) have been suggested. They replace unknown terms in the differential equations with ANN. Such hybrid models allow to both encode prior domain knowledge, such as first principles, and to learn unknown mechanisms from data. Often, data for the training of UDE can only be obtained via costly experiments. We consider optimal experimental design (OED) for planning of experiments and generating data needed to train UDE. The number of weights in the embedded ANN usually leads to an overfitting of the regression problem. To make the OED problem tractable for optimization, we propose and compare dimension reduction methods that are based on lumping of weights and singular value decomposition of the Fisher information matrix (FIM), respectively. They result in lower-dimensional variational differential equations, which are easier to solve and yield regular FIM. Our numerical results showcase the advantages of OED for UDE, such as increased data-efficiency and better extrapolation properties.

Single-cell transcriptomics, epigenomics, and other 'omics applied at single-cell resolution can significantly advance hypotheses and understanding of glial biology. Omics technologies are revealing a large and growing number of new glial cell subtypes, defined by their gene expression profile. These subtypes have significant implications for understanding glial cell function, cell-cell communications, and glia-specific changes between homeostasis and conditions such as neurological disease. For many, the training in how to analyze, interpret, and understand these large datasets has been through reading and understanding literature from other fields like biostatistics. Here, we provide a primer for glial biologists on experimental design and analysis of single-cell RNA-seq datasets. Our goal is to further the understanding of why decisions might be made about datasets and to enhance biologists' ability to interpret and critique their work and the work of others. We review the steps involved in single-cell analysis with a focus on decision points and particular notes for glia. The goal of this primer is to ensure that single-cell 'omics experiments continue to advance glial biology in a rigorous and replicable way., Comment: 66 pages, 1 table, 5 figures

The exploration of multicomponent material composition space requires significant time and financial investments, necessitating efficient use of resources for statistically relevant compositions. This article introduces a novel methodology, implemented in the open-source CASTRO (ConstrAined Sequential laTin hypeRcube sampling methOd) software package, to overcome equality-mixture constraints and ensure comprehensive design space coverage. Our approach leverages Latin hypercube sampling (LHS) and LHS with multidimensional uniformity (LHSMDU) using a divide-and-conquer strategy to manage high-dimensional problems effectively. By incorporating previous experimental knowledge within a limited budget, our method strategically recommends a feasible number of experiments to explore the design space. We validate our methodology with two examples: a four-dimensional problem with near-uniform distributions and a nine-dimensional problem with additional mixture constraints, yielding specific component distributions. Our constrained sequential LHS or LHSMDU approach enables thorough design space exploration, proving robustness for experimental design. This research not only advances material science but also offers promising solutions for efficiency challenges in pharmaceuticals and chemicals. CASTRO and the case studies are available for free download on GitHub., Comment: 35 pages, 11 figures, 2 tables

Questions of `how best to acquire data' are essential to modeling and prediction in the natural and social sciences, engineering applications, and beyond. Optimal experimental design (OED) formalizes these questions and creates computational methods to answer them. This article presents a systematic survey of modern OED, from its foundations in classical design theory to current research involving OED for complex models. We begin by reviewing criteria used to formulate an OED problem and thus to encode the goal of performing an experiment. We emphasize the flexibility of the Bayesian and decision-theoretic approach, which encompasses information-based criteria that are well-suited to nonlinear and non-Gaussian statistical models. We then discuss methods for estimating or bounding the values of these design criteria; this endeavor can be quite challenging due to strong nonlinearities, high parameter dimension, large per-sample costs, or settings where the model is implicit. A complementary set of computational issues involves optimization methods used to find a design; we discuss such methods in the discrete (combinatorial) setting of observation selection and in settings where an exact design can be continuously parameterized. Finally we present emerging methods for sequential OED that build non-myopic design policies, rather than explicit designs; these methods naturally adapt to the outcomes of past experiments in proposing new experiments, while seeking coordination among all experiments to be performed. Throughout, we highlight important open questions and challenges., Comment: Appears in Acta Numerica 2024. This version contains an evolving set of post-publication additions and corrections

Miniaturized magnetic soft robots have shown extraordinary capabilities of contactless manipulation, complex path maneuvering, precise localization, and quick actuation, which have equipped them to cater to challenging biomedical applications such as targeted drug delivery, internal wound healing, and laparoscopic surgery. However, despite their successful fabrication by several different research groups, a thorough design strategy encompassing the optimized kinematic performance of the three fundamental biomimetic swimming modes at miniaturized length scales has not been reported till now. Here, we resolve this by designing magnetic soft robotic swimmers (MSRSs) from the class of helical and undulatory low Reynolds number (Re) swimmers using a fully coupled, experimentally calibrated computational fluid dynamics model. We study (and compare) their swimming performance, and report their steady-state swimming speed for different non-dimensional numbers that capture the competition by magnetic loading, non-linear elastic deformation and viscous solid-fluid coupling. We investigate their stability for different initial spatial orientations to ensure robustness during real-life applications. Our results show that the helical 'finger-shaped' swimmer is, by far, the fastest low Re swimmer in terms of body lengths per cycle, but that the undulatory 'carangiform' swimmer proved to be the most versatile, bi-directional swimmer with maximum stability.

Optimal experimental design is a well studied field in applied science and engineering. Techniques for estimating such a design are commonly used within the framework of parameter estimation. Nonetheless, in recent years parameter estimation techniques are changing rapidly with the introduction of deep learning techniques to replace traditional estimation methods. This in turn requires the adaptation of optimal experimental design that is associated with these new techniques. In this paper we investigate a new experimental design methodology that uses deep learning. We show that the training of a network as a Likelihood Free Estimator can be used to significantly simplify the design process and circumvent the need for the computationally expensive bi-level optimization problem that is inherent in optimal experimental design for non-linear systems. Furthermore, deep design improves the quality of the recovery process for parameter estimation problems. As proof of concept we apply our methodology to two different systems of Ordinary Differential Equations.

We consider optimal experimental design (OED) for nonlinear inverse problems within the Bayesian framework. Optimizing the data acquisition process for large-scale nonlinear Bayesian inverse problems is a computationally challenging task since the posterior is typically intractable and commonly-encountered optimality criteria depend on the observed data. Since these challenges are not present in OED for linear Bayesian inverse problems, we propose an approach based on first linearizing the associated forward problem and then optimizing the experimental design. Replacing an accurate but costly model with some linear surrogate, while justified for certain problems, can lead to incorrect posteriors and sub-optimal designs if model discrepancy is ignored. To avoid this, we use the Bayesian approximation error (BAE) approach to formulate an A-optimal design objective for sensor selection that is aware of the model error. In line with recent developments, we prove that this uncertainty-aware objective is independent of the exact choice of linearization. This key observation facilitates the formulation of an uncertainty-aware OED objective function using a completely trivial linear map, the zero map, as a surrogate to the forward dynamics. The base methodology is also extended to marginalized OED problems, accommodating uncertainties arising from both linear approximations and unknown auxiliary parameters. Our approach only requires parameter and data sample pairs, hence it is particularly well-suited for black box forward models. We demonstrate the effectiveness of our method for finding optimal designs in an idealized subsurface flow inverse problem and for tsunami detection.

Experimental design techniques such as active search and Bayesian optimization are widely used in the natural sciences for data collection and discovery. However, existing techniques tend to favor exploitation over exploration of the search space, which causes them to get stuck in local optima. This ``collapse" problem prevents experimental design algorithms from yielding diverse high-quality data. In this paper, we extend the Vendi scores -- a family of interpretable similarity-based diversity metrics -- to account for quality. We then leverage these quality-weighted Vendi scores to tackle experimental design problems across various applications, including drug discovery, materials discovery, and reinforcement learning. We found that quality-weighted Vendi scores allow us to construct policies for experimental design that flexibly balance quality and diversity, and ultimately assemble rich and diverse sets of high-performing data points. Our algorithms led to a 70%-170% increase in the number of effective discoveries compared to baselines., Comment: Published in International Conference on Machine Learning, ICML 2024. Code can be found in the Vertaix GitHub: https://github.com/vertaix/Quality-Weighted-Vendi-Score. Paper dedicated to Kwame Nkrumah

We applied Bayesian Optimal Experimental Design (OED) in the estimation of parameters involved in the Equilibrium Dispersive Model for chromatography with two components with the Langmuir adsorption isotherm. The coefficients estimated were Henry's coefficients, the total absorption capacity and the number of theoretical plates, while the design variables were the injection time and the initial concentration. The Bayesian OED algorithm is based on nested Monte Carlo estimation, which becomes computationally challenging due to the simulation time of the PDE involved in the dispersive model. This complication was relaxed by introducing a surrogate model based on Piecewise Sparse Linear Interpolation. Using the surrogate model instead the original reduces significantly the simulation time and it approximates the solution of the PDE with high degree of accuracy. The estimation of the parameters over strategical design points provided by OED reduces the uncertainty in the estimation of parameters. Additionally, the Bayesian OED methodology indicates no improvements when increasing the number of measurements in temporal nodes above a threshold value., Comment: 23 pages and 8 figures

We develop adaptive discretization algorithms for locally optimal experimental design of nonlinear prediction models. With these algorithms, we refine and improve a pertinent state-of-the-art algorithm in various respects. We establish novel termination, convergence, and convergence rate results for the proposed algorithms. In particular, we prove a sublinear convergence rate result under very general assumptions on the design criterion and, most notably, a linear convergence result under the additional assumption that the design criterion is strongly convex and the design space is finite. Additionally, we prove the finite termination at approximately optimal designs, including upper bounds on the number of iterations until termination. And finally, we illustrate the practical use of the proposed algorithms by means of two application examples from chemical engineering: one with a stationary model and one with a dynamic model., Comment: 42 pages

Accurate parameter dependent electro-chemical numerical models for lithium-ion batteries are essential in industrial application. The exact parameters of each battery cell are unknown and a process of estimation is necessary to infer them. The parameter estimation generates an accurate model able to reproduce real cell data. The field of optimal input/experimental design deals with creating the experimental settings facilitating the estimation problem. Here we apply two different input design algorithms that aim at maximizing the observability of the true, unknown parameters: in the first algorithm, we design the applied current and the starting voltage. This lets the algorithm collect information on different states of charge, but requires long experimental times (60 000 s). In the second algorithm, we generate a continuous current, composed of concatenated optimal intervals. In this case, the experimental time is shorter (7000 s) and numerical experiments with virtual data give an even better accuracy results, but experiments with real battery data reveal that the accuracy could decrease hundredfold. As the design algorithms are built independent of the model, the same results and motivation are applicable to more complex battery cell models and, moreover, to other applications., Comment: 29 pages, 8 figures

Bayesian optimal experimental design (OED) seeks experiments that maximize the expected information gain (EIG) in model parameters. Directly estimating the EIG using nested Monte Carlo is computationally expensive and requires an explicit likelihood. Variational OED (vOED), in contrast, estimates a lower bound of the EIG without likelihood evaluations by approximating the posterior distributions with variational forms, and then tightens the bound by optimizing its variational parameters. We introduce the use of normalizing flows (NFs) for representing variational distributions in vOED; we call this approach vOED-NFs. Specifically, we adopt NFs with a conditional invertible neural network architecture built from compositions of coupling layers, and enhanced with a summary network for data dimension reduction. We present Monte Carlo estimators to the lower bound along with gradient expressions to enable a gradient-based simultaneous optimization of the variational parameters and the design variables. The vOED-NFs algorithm is then validated in two benchmark problems, and demonstrated on a partial differential equation-governed application of cathodic electrophoretic deposition and an implicit likelihood case with stochastic modeling of aphid population. The findings suggest that a composition of 4--5 coupling layers is able to achieve lower EIG estimation bias, under a fixed budget of forward model runs, compared to previous approaches. The resulting NFs produce approximate posteriors that agree well with the true posteriors, able to capture non-Gaussian and multi-modal features effectively.

We conducted an eye-tracking user study with 13 participants to investigate the influence of stimulus-question ordering and question modality on participants using visual question-answering (VQA) tasks. We examined cognitive load, task performance, and gaze allocations across five distinct experimental designs, aiming to identify setups that minimize the cognitive burden on participants. The collected performance and gaze data were analyzed using quantitative and qualitative methods. Our results indicate a significant impact of stimulus-question ordering on cognitive load and task performance, as well as a noteworthy effect of question modality on task performance. These findings offer insights for the experimental design of controlled user studies in visualization research., Comment: Accepted at ETVIS 2024

We introduce a new method to jointly reduce the dimension of the input and output space of a high-dimensional function. Choosing a reduced input subspace influences which output subspace is relevant and vice versa. Conventional methods focus on reducing either the input or output space, even though both are often reduced simultaneously in practice. Our coupled approach naturally supports goal-oriented dimension reduction, where either an input or output quantity of interest is prescribed. We consider, in particular, goal-oriented sensor placement and goal-oriented sensitivity analysis, which can be viewed as dimension reduction where the most important output or, respectively, input components are chosen. Both applications present difficult combinatorial optimization problems with expensive objectives such as the expected information gain and Sobol indices. By optimizing gradient-based bounds, we can determine the most informative sensors and most sensitive parameters as the largest diagonal entries of some diagnostic matrices, thus bypassing the combinatorial optimization and objective evaluation.

The stable unit treatment value assumption states that the outcome of an individual is not affected by the treatment statuses of others, however in many real world applications, treatments can have an effect on many others beyond the immediately treated. Interference can generically be thought of as mediated through some network structure. In many empirically relevant situations however, complete network data (required to adjust for these spillover effects) are too costly or logistically infeasible to collect. Partially or indirectly observed network data (e.g., subsamples, aggregated relational data (ARD), egocentric sampling, or respondent-driven sampling) reduce the logistical and financial burden of collecting network data, but the statistical properties of treatment effect adjustments from these design strategies are only beginning to be explored. In this paper, we present a framework for the estimation and inference of treatment effect adjustments using partial network data through the lens of structural causal models. We also illustrate procedures to assign treatments using only partial network data, with the goal of either minimizing estimator variance or optimally seeding. We derive single network asymptotic results applicable to a variety of choices for an underlying graph model. We validate our approach using simulated experiments on observed graphs with applications to information diffusion in India and Malawi.

Causal discovery aims to uncover cause-and-effect relationships encoded in causal graphs by leveraging observational, interventional data, or their combination. The majority of existing causal discovery methods are developed assuming infinite interventional data. We focus on data interventional efficiency and formalize causal discovery from the perspective of online learning, inspired by pure exploration in bandit problems. A graph separating system, consisting of interventions that cut every edge of the graph at least once, is sufficient for learning causal graphs when infinite interventional data is available, even in the worst case. We propose a track-and-stop causal discovery algorithm that adaptively selects interventions from the graph separating system via allocation matching and learns the causal graph based on sampling history. Given any desired confidence value, the algorithm determines a termination condition and runs until it is met. We analyze the algorithm to establish a problem-dependent upper bound on the expected number of required interventional samples. Our proposed algorithm outperforms existing methods in simulations across various randomly generated causal graphs. It achieves higher accuracy, measured by the structural hamming distance (SHD) between the learned causal graph and the ground truth, with significantly fewer samples., Comment: To appear in Proceedings of ICML 24

In cancer research, leveraging patient-derived xenografts (PDXs) in pre-clinical experiments is a crucial approach for assessing innovative therapeutic strategies. Addressing the inherent variability in treatment response among and within individual PDX lines is essential. However, the current literature lacks a user-friendly statistical power analysis tool capable of concurrently determining the required number of PDX lines and animals per line per treatment group in this context. In this paper, we present a simulation-based R package for sample size determination, named `\textbf{PDXpower}', which is publicly available at The Comprehensive R Archive Network \url{https://CRAN.R-project.org/package=PDXpower}. The package is designed to estimate the necessary number of both PDX lines and animals per line per treatment group for the design of a PDX experiment, whether for an uncensored outcome, or a censored time-to-event outcome. Our sample size considerations rely on two widely used analytical frameworks: the mixed effects ANOVA model for uncensored outcomes and Cox's frailty model for censored data outcomes, which effectively account for both inter-PDX variability and intra-PDX correlation in treatment response. Step-by-step illustrations for utilizing the developed package are provided, catering to scenarios with or without preliminary data.

One emergent ability of large language models (LLMs) is that query-specific examples can be included in the prompt at inference time. In this work, we use active learning for adaptive prompt design and call it Active In-context Prompt Design (AIPD). We design the LLM prompt by adaptively choosing few-shot examples from a training set to optimize performance on a test set. The training examples are initially unlabeled and we obtain the label of the most informative ones, which maximally reduces uncertainty in the LLM prediction. We propose two algorithms, GO and SAL, which differ in how the few-shot examples are chosen. We analyze these algorithms in linear models: first GO and then use its equivalence with SAL. We experiment with many different tasks in small, medium-sized, and large language models; and show that GO and SAL outperform other methods for choosing few-shot examples in the LLM prompt at inference time.