Search

A common feature of computational modelling and simulation research is the need to perform many tasks in complex sequences to achieve a usable result. This will typically involve tasks such as preparing input data, pre-processing, running simulations on a local or remote machine, post-processing, and performing coupling communications, validations and/or optimisations. Tasks like these can involve manual steps which are time and effort intensive, especially when it involves the management of large ensemble runs. Additionally, human errors become more likely and numerous as the research work becomes more complex, increasing the risk of damaging the credibility of simulation results. Automation tools can help ensure the credibility of simulation results by reducing the manual time and effort required to perform these research tasks, by making more rigorous procedures tractable, and by reducing the probability of human error due to a reduced number of manual actions. In addition, efficiency gained through automation can help researchers to perform more research within the budget and effort constraints imposed by their projects. This paper presents the main software release of FabSim3, and explains how our automation toolkit can improve and simplify a range of tasks for researchers and application developers. FabSim3 helps to prepare, submit, execute, retrieve, and analyze simulation workflows. By providing a suitable level of abstraction, FabSim3 reduces the complexity of setting up and managing a large-scale simulation scenario, while still providing transparent access to the underlying layers for effective debugging. The tool also facilitates job submission and management (including staging and curation of files and environments) for a range of different supercomputing environments. Although FabSim3 itself is application-agnostic, it supports a provably extensible plugin system where users automate simulation and analysis workflows for their own application domains. To highlight this, we briefly describe a selection of these plugins and we demonstrate the efficiency of the toolkit in handling large ensemble workflows. EPSRC under grant agreement EP/W007711/1, as well as by the VECMA and HiDALGO projects, which have received funding from the European Union Horizon 2020 research and innovation programme under grant agreement nos 800925 and 824115. In addition, FabFlee was supported by the ITFLOWS project and FabCovid19 by the STAMINA project, both of which have received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 882986 and No 883441 respectively

Lockdowns and associated measures imposed in response to the COVID-19 crisis inflict severe damage to society. Across the globe, scientists and policymakers study ways to lift measures while maintaining control of virus spread in circumstances that continuously change due to the evolution of new variants and increasing vaccination coverage. In this process, it has become clear that finding and analysing exit strategies, which are a key aspect of pandemic mitigation in all consecutive waves of infection, is not solely a matter of epidemiological modeling but has many different dimensions that need to be balanced and therefore requires input from many different disciplines. Here, we document an attempt to investigate exit strategies from a multidisciplinary perspective through the Science versus Corona project in the Netherlands. In this project, scientists and laypeople were challenged to submit (components of) exit strategies. A selection of these were implemented in a formal model, and we have evaluated the scenarios from a multidisciplinary perspective, utilizing expertise in epidemiology, economics, psychology, law, mathematics, and history. We argue for the integration of multidisciplinary perspectives on COVID-19 and more generally in pandemic mitigation, highlight open challenges, and present an agenda for further research into exit strategies and their assessment.

Many countries are currently dealing with the COVID-19 epidemic and are searching for an exit strategy such that life in society can return to normal. To support this search, computational models are used to predict the spread of the virus and to assess the efficacy of policy measures before actual implementation. The model output has to be interpreted carefully though, as computational models are subject to uncertainties. These can stem from, e.g., limited knowledge about input parameters values or from the intrinsic stochastic nature of some computational models. They lead to uncertainties in the model predictions, raising the question what distribution of values the model produces for key indicators of the severity of the epidemic. Here we show how to tackle this question using techniques for uncertainty quantification and sensitivity analysis. We assess the uncertainties and sensitivities of four exit strategies implemented in an agent-based transmission model with geographical stratification. The exit strategies are termed Flattening the Curve, Contact Tracing, Intermittent Lockdown and Phased Opening. We consider two key indicators of the ability of exit strategies to avoid catastrophic health care overload: the maximum number of prevalent cases in intensive care (IC), and the total number of IC patient-days in excess of IC bed capacity. Our results show that uncertainties not directly related to the exit strategies are secondary, although they should still be considered in comprehensive analysis intended to inform policy makers. The sensitivity analysis discloses the crucial role of the intervention uptake by the population and of the capability to trace infected individuals. Finally, we explore the existence of a safe operating space. For Intermittent Lockdown we find only a small region in the model parameter space where the key indicators of the model stay within safe bounds, whereas this region is larger for the other exit strategies., Author summary Many countries are currently dealing with the COVID-19 epidemic and are looking for an exit strategy such that life in society can return to normal. For that purpose computational models are used to predict the spread of the virus and to assess the efficacy of policy measures before putting them into practice. These models are subject to uncertainties (due to, for instance, limited knowledge of the parameter values), which can lead to a large variability in model predictions. It is therefore fundamental to assess which range of values a model produces for key indicators of the severity of the epidemic. We present here the results of the uncertainty and sensitivity analysis of four exit strategies simulated with an individual-based model of the COVID-19 transmission. As key indicators of the severity of the pandemic we consider the maximum number of cases in intensive care and the total number of intensive care patient-days in excess. Our results show the crucial role of the intervention uptake by the population, of the reduction in the level of transmission by intervention and of the capability to trace infected individuals.

Classical molecular dynamics is a computer simulation technique that is in widespread use across many areas of science, from physics and chemistry to materials, biology, and medicine. The method continues to attract criticism due its oft-reported lack of reproducibility which is in part due to a failure to submit it to reliable uncertainty quantification (UQ). Here we show that the uncertainty arises from a combination of (i) the input parameters and (ii) the intrinsic stochasticity of the method controlled by the random seeds. To illustrate the situation, we make a systematic UQ analysis of a widely used molecular dynamics code (NAMD), applied to estimate binding free energy of a ligand-bound to a protein. In particular, we replace the usually fixed input parameters with random variables, systematically distributed about their mean values, and study the resulting distribution of the simulation output. We also perform a sensitivity analysis, which reveals that, out of a total of 175 parameters, just six dominate the variance in the code output. Furthermore, we show that binding energy calculations dampen the input uncertainty, in the sense that the variation around the mean output free energy is less than the variation around the mean of the assumed input distributions, if the output is ensemble-averaged over the random seeds. Without such ensemble averaging, the predicted free energy is five times more uncertain. The distribution of the predicted properties is thus strongly dependent upon the random seed. Owing to this substantial uncertainty, robust statistical measures of uncertainty in molecular dynamics simulation require the use of ensembles in all contexts.

In simulations of multiscale dynamical systems, not all relevant processes can be resolved explicitly. Taking the effect of the unresolved processes into account is important, which introduces the need for paramerizations. We present a machine-learning method, used for the conditional resampling of observations or reference data from a fully resolved simulation. It is based on the probabilistic classiffcation of subsets of reference data, conditioned on macroscopic variables. This method is used to formulate a parameterization that is stochastic, taking the uncertainty of the unresolved scales into account. We validate our approach on the Lorenz 96 system, using two different parameter settings which are challenging for parameterization methods., 27 pages, 17 figures. Submitted

Copyright © 2021 The Author(s). The VECMA toolkit enables automated Verification, Validation and Uncertainty Quantification (VVUQ) for complex applications that can be deployed on emerging exascale platforms and provides support for software applications for any domain of interest. The toolkit has four main components including EasyVVUQ for VVUQ workflows, FabSim3 for automation and tool integration, MUSCLE3 for coupling multiscale models and QCG tools to execute application workflows on high performance computing (HPC). A more recent addition to the VECMAtk is EasySurrogate for various types of surrogate methods. In this paper, we present five tutorials from different application domains that apply these VECMAtk components to perform uncertainty quantification analysis, use surrogate models, couple multiscale models and execute sensitivity analysis on HPC. This paper aims to provide hands-on experience for practitioners aiming to test and contrast with their own applications. This work was supported by the VECMA project, which has received funding from the European Union Horizon 2020 research and innovation programme under grant agreement No. 800925. The development of MUSCLE3 and its respective description was supported by the Netherlands eScience Center and NWO under the e-MUSC project. The development of ISR3D was supported by the InSilc project and the In Silico World (ISW) project (European Union Horizon 2020 research and innovation programme grant agreements #777119 and #101016503 respectively). The calculations were performed in the Poznan Supercomputing and Networking Center.

We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantification (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year, we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail., 23 pages, 10 figures

Uncertainty quantification (UQ) is a key component when using computational models that involve uncertainties, e.g. in decision-making scenarios. In this work, we present uncertainty quantification patterns (UQPs) that are designed to support the analysis of uncertainty in coupled multi-scale and multi-domain applications. UQPs provide the basic building blocks to create tailored UQ for multiscale models. The UQPs are implemented as generic templates, which can then be customized and aggregated to create a dedicated UQ procedure for multiscale applications. We present the implementation of the UQPs with multiscale coupling toolkit Multiscale Coupling Library and Environment 3. Potential speed-up for UQPs has been derived as well. As a proof of concept, two examples of multiscale applications using UQPs are presented.This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantificationin silico’.

In this study, we investigate uncertainties in a large eddy simulation of the atmosphere, employing modern uncertainty quantification methods that have hardly been used yet in this context. When analysing the uncertainty of model results, one can distinguish between uncertainty related to physical parameters whose values are not exactly known, and uncertainty related to modelling choices such as the selection of numerical discretization methods, of the spatial domain size and resolution, and the use of different model formulations. While the former kind is commonly studied e.g. with forward uncertainty propagation, we explore the use of such techniques to also assess the latter kind. From a climate modelling perspective, uncertainties in the convective response and cloud formation are of particular interest, since these affect the cloud-climate feedback, one of the dominant sources of uncertainty in current climate models. Therefore we analyse the DALES model in the RICO case, a well-studied convection benchmark. We use the VECMA toolkit for uncertainty propagation, assessing uncertainties stemming from physical parameters as well as from modelling choices. We find substantial uncertainties due to small random initial state perturbations, and that the choice of advection scheme is the most influential of the modelling choices we assessed. This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.

It is well known that the wide range of spatial and temporal scales present in geophysical flow problems represents a (currently) insurmountable computational bottleneck, which must be circumvented by a coarse-graining procedure. The effect of the unresolved fluid motions enters the coarse-grained equations as an unclosed forcing term, denoted as the ’eddy forcing’. Traditionally, the system is closed by approximate deterministic closure models, i.e. so-called parameterizations. Instead of creating a deterministic parameterization, some recent efforts have focused on creating a stochastic, data-driven surrogate model for the eddy forcing from a (limited) set of reference data, with the goal of accurately capturing the long-term flow statistics. Since the eddy forcing is a dynamically evolving field, a surrogate should be able to mimic the complex spatial patterns displayed by the eddy forcing. Rather than creating such a (fully data-driven) surrogate, we propose to precede the surrogate construction step by a procedure that replaces the eddy forcing with a new source term which: i) is tailor-made to capture spatially-integrated quantities of interest, ii) strikes a balance between physical insight and data-driven modelling , and iii) significantly reduces the amount of training data that is needed. Instead of creating a surrogate model for an evolving field, we now only require a surrogate model for one scalar time series per quantity-of-interest. We derive the new source terms for a simplified an ocean model of two-dimensional turbulence in a doubly periodic square domain, and show that the time-series training data produces the same statistics for our quantities of interest as the full-field eddy-forcing.

Data availability Figure 1a,b displays publicly available cumulative death count data for the UK, which were obtained from ref. 22. Source Data are available with this paper. Furthermore, the parameter list—with all input parameters, a description, their default values and reasons for inclusion or exclusion from the Imperial College London CovidSim team—is available as Supplementary Data. Supplementary information: https://static-content.springer.com/esm/art%3A10.1038%2Fs43588-021-00028-9/MediaObjects/43588_2021_28_MOESM1_ESM.pdf Supplementary discussion, Figs. 1–7 and Table 1. Supplementary Data 1: https://static-content.springer.com/esm/art%3A10.1038%2Fs43588-021-00028-9/MediaObjects/43588_2021_28_MOESM2_ESM.xlsx All input parameters, a description, their default values and reasons for inclusion or exclusion from the Imperial College London CovidSim team. Source data Source Data Fig. 1: https://static-content.springer.com/esm/art%3A10.1038%2Fs43588-021-00028-9/MediaObjects/43588_2021_28_MOESM3_ESM.xlsx The computed cumulative death results for Fig. 1 and the observed death count data. Source Data Fig. 2: https://static-content.springer.com/esm/art%3A10.1038%2Fs43588-021-00028-9/MediaObjects/43588_2021_28_MOESM4_ESM.xlsx The Sobol indices of Fig. 2. Copyright © 2022 The Autor(s). Epidemiological modelling has assisted in identifying interventions that reduce the impact of COVID-19. The UK government relied, in part, on the CovidSim model to guide its policy to contain the rapid spread of the COVID-19 pandemic during March and April 2020; however, CovidSim contains several sources of uncertainty that affect the quality of its predictions: parametric uncertainty, model structure uncertainty and scenario uncertainty. Here we report on parametric sensitivity analysis and uncertainty quantification of the code. From the 940 parameters used as input into CovidSim, we find a subset of 19 to which the code output is most sensitive—imperfect knowledge of these inputs is magnified in the outputs by up to 300%. The model displays substantial bias with respect to observed data, failing to describe validation data well. Quantifying parametric input uncertainty is therefore not sufficient: the effect of model structure and scenario uncertainty must also be properly understood. Royal Society’s RAMP initiative: European Union Horizon 2020 research and innovation programme under grant agreement nos. 800925 (VECMA project; www.vecma.eu) and 823712 (CompBioMed2 Centre of Excellence, www.compbiomed.eu); UK EPSRC for the UK High-End Computing Consortium (grant no. EP/R029598/1).

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV2) virus has rapidly spread worldwide since December 2019, and early modelling work of this pandemic has assisted in identifying effective government interventions. The UK government relied in part on the CovidSim model developed by the MRC Centre for Global Infectious Disease Analysis at Imperial College London, to model various non-pharmaceutical intervention strategies, and guide its government policy in seeking to deal with the rapid spread of the COVID-19 pandemic during March and April 2020. CovidSim is subject to different sources of uncertainty, namely parametric uncertainty in the inputs, model structure uncertainty (i.e., missing epidemiological processes) and scenario uncertainty, which relates to uncertainty in the set of conditions under which the model is applied. We have undertaken an extensive parametric sensitivity analysis and uncertainty quantification of the current CovidSim code. From the over 900 parameters that are provided as input to CovidSim, we identified a key subset of 19 parameters to which the code output is most sensitive. We find that the uncertainty in the code is substantial, in the sense that imperfect knowledge in these inputs will be magnified to the outputs, up to the extent of ca. 300%. Most of this uncertainty can be traced back to the sensitivity of three parameters. Compounding this, the model can display significant bias with respect to observed data, such that the output variance does not capture this validation data with high probability. We conclude that quantifying the parametric input uncertainty is not sufficient, and that the effect of model structure and scenario uncertainty cannot be ignored when validating the model in a probabilistic sense.

© 2020 The Authors. Validation, verification, and uncertainty quantification (VVUQ) of simulation workflows are essential for building trust in simulation results, and their increased use in decision‐making processes. The EasyVVUQ Python library is designed to facilitate implementation of advanced VVUQ techniques in new or existing workflows, with a particular focus on high‐performance computing, middleware agnosticism, and multiscale modeling. Here, the application of EasyVVUQ to five very diverse application areas is demonstrated: materials properties, ocean circulation modeling, fusion reactors, forced human migration, and urban air quality prediction. Horizon 2020 Framework Programme. Grant Number: 800925; Engineering and Physical Sciences Research Council

Recent years have seen a growing interest in using data-driven (machine-learning) techniques for the construction of cheap surrogate models of turbulent subgrid scale stresses. These stresses display complex spatio-temporal structures, and constitute a difficult surrogate target. In this paper we propose a data-preprocessing step, in which we derive alternative subgrid scale models which are virtually exact for a user-specified set of spatially integrated quantities of interest. The unclosed component of these new subgrid scale models is of the same size as this set of integrated quantities of interest. As a result, the corresponding training data is massively reduced in size, decreasing the complexity of the subsequent surrogate construction.

Reynolds-averaged Navier-Stokes (RANS) models are the primary numerical recourse to investigate complex engineering turbulent flows in industrial applications. However, to establish RANS models as reliable design tools, it is essential to provide estimates for the uncertainty in their predictions. In the recent past, an uncertainty estimation framework relying on eigenvalue and eigenvector perturbations to the modeled Reynolds stress tensor has been widely applied with satisfactory results. However, the methodology for the eigenvector perturbations is not well established. Evaluations using only eigenvalue perturbations do not provide comprehensive estimates of model form uncertainty, especially in flows with streamline curvature, recirculation, or flow separation. In this article, we outline a methodology for the eigenvector perturbations using a predictor-corrector approach, which uses the incipient eigenvalue perturbations along with the Reynolds stress transport equations to determine the eigenvector perturbations. This approach was applied to benchmark cases of complex turbulent flows. The uncertainty intervals estimated using the proposed framework exhibited substantial improvement over eigenvalue-only perturbations and are able to account for a significant proportion of the discrepancy between RANS predictions and high-fidelity data.

It is well known that the wide range of spatial and temporal scales present in geophysical flow problems represents a (currently) insurmountable computational bottleneck, which must be circumvented by a coarse-graining procedure. The effect of the unresolved fluid motions enters the coarse-grained equations as an unclosed forcing term, denoted as the ’eddy forcing’. Traditionally, the system is closed by approximate deterministic closure models, i.e. so-called parameterizations. Instead of creating a deterministic parameterization, some recent efforts have focused on creating a stochastic, data-driven surrogate model for the eddy forcing from a (limited) set of reference data, with the goal of accurately capturing the long-term flow statistics. Since the eddy forcing is a dynamically evolving field, a surrogate should be able to mimic the complex spatial patterns displayed by the eddy forcing. Rather than creating such a (fully data-driven) surrogate, we propose to precede the surrogate construction step by a proce- dure that replaces the eddy forcing with a new model-error source term which: i) is tailor-made to capture spatially-integrated statistics of interest, ii) strikes a balance between physical in- sight and data-driven modelling , and iii) significantly reduces the amount of training data that is needed. Instead of creating a surrogate for an evolving field, we now only require a surrogate model for one scalar time series per statistical quantity-of-interest. Our current surrogate mod- elling approach builds on a resampling strategy, where we create a probability density function of the reduced training data that is conditional on (time-lagged) resolved-scale variables. We derive the model-error source terms, and construct the reduced surrogate using an ocean model of two-dimensional turbulence in a doubly periodic square domain.

EasyVVUQ is an open source Python library (https://github.com/UCL-CCS/EasyVVUQ) designed to facilitate verification, validation and uncertainty quantification (VVUQ) for a wide variety of simulations. The goal of EasyVVUQ is to make it as easy as possible to implement advanced VVUQ techniques for existing application codes or workflows. Our aim is to expose these features in an accessible way for users of scientific software, in particular for simulation codes running on high performance computers. Funding statement: We acknowledge funding support from the European Union’s Horizon 2020 research and innovation programme under grant agreement 800925 (VECMA project, www.vecma.eu) and the UK Consortium on Mesoscale Engineering Sciences (UK-COMES, http://www.ukcomes.org), EPSRC reference EP/L00030X/1.