Search

Background: Mathematical models served a critical role in COVID-19 decision making throughout the pandemic. Model calibration is an essential, but often computationally burdensome, step in model development that provides estimates for difficult-to-measure parameters and establishes an up-to-date modeling platform for scenario analysis. In the evolving COVID-19 pandemic, frequent recalibration was necessary to provide ongoing support to decision makers. In this study, we address the computational challenges of frequent recalibration with a new calibration approach., Methods: We calibrated and recalibrated an age-stratified dynamic compartmental model of COVID-19 in Minnesota to statewide COVID-19 cumulative mortality and prevalent age-specific hospitalizations from March 22, 2020 through August 20, 2021. This period was divided into 10 calibration periods, reflecting significant changes in policies, messaging, and/or epidemiological conditions in Minnesota. When recalibrating the model from one period to the next, we employed a sequential calibration approach that leveraged calibration results from previous periods and adjusted only parameters most relevant to the calibration target data of the new calibration period to improve computational efficiency. We compared computational burden and performance of the sequential calibration approach to a more traditional calibration method, in which all parameters were readjusted with each recalibration., Results: Both calibration methods identified parameter sets closely reproducing prevalent hospitalizations and cumulative deaths over time. By the last calibration period, both approaches converged to similar parameter values. However, the sequential calibration approach identified parameter sets that more tightly fit calibration targets and required substantially less computation time than traditional calibration., Conclusions: Sequential calibration is an efficient approach to maintaining up-to-date models with evolving, time-varying parameters and potentially identifies better-fitting parameter sets than traditional calibration., Highlights: This study used a sequential calibration approach, which takes advantage of previous calibration results to reduce the number of parameters to be estimated in each round of calibration, improving computational efficiency and algorithm convergence to best-fitting parameter values.Both sequential and traditional calibration approaches were able to identify parameter sets that closely reproduced calibration targets. However, the sequential calibration approach generated parameter sets that yielded tighter fits and was less computationally burdensome.Sequential calibration is an efficient approach to maintaining up-to-date models with evolving, time-varying parameters., Competing Interests: The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The following authors are employed by the sponsor: ECS, ABS, PJM, SG. EAE received personal consulting fees from Janssen Pharmaceuticals for work unrelated to this study. ZL, SBM, SYZK, KMK have no conflicts of interest to report. The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this study was provided entirely by a contract from the Minnesota Department of Health. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report

Decision models can combine information from different sources to simulate the long-term consequences of alternative strategies in the presence of uncertainty. A cohort state-transition model (cSTM) is a decision model commonly used in medical decision making to simulate the transitions of a hypothetical cohort among various health states over time. This tutorial focuses on time-independent cSTM, in which transition probabilities among health states remain constant over time. We implement time-independent cSTM in R, an open-source mathematical and statistical programming language. We illustrate time-independent cSTMs using a previously published decision model, calculate costs and effectiveness outcomes, and conduct a cost-effectiveness analysis of multiple strategies, including a probabilistic sensitivity analysis. We provide open-source code in R to facilitate wider adoption. In a second, more advanced tutorial, we illustrate time-dependent cSTMs.

In an introductory tutorial, we illustrated building cohort state-transition models (cSTMs) in R, where the state transition probabilities were constant over time. However, in practice, many cSTMs require transitions, rewards, or both to vary over time (time dependent). This tutorial illustrates adding 2 types of time dependence using a previously published cost-effectiveness analysis of multiple strategies as an example. The first is simulation-time dependence, which allows for the transition probabilities to vary as a function of time as measured since the start of the simulation (e.g., varying probability of death as the cohort ages). The second is state-residence time dependence, allowing for history by tracking the time spent in any particular health state using tunnel states. We use these time-dependent cSTMs to conduct cost-effectiveness and probabilistic sensitivity analyses. We also obtain various epidemiological outcomes of interest from the outputs generated from the cSTM, such as survival probability and disease prevalence, often used for model calibration and validation. We present the mathematical notation first, followed by the R code to execute the calculations. The full R code is provided in a public code repository for broader implementation.

Background: Analyses of the effectiveness of infectious disease control interventions often rely on dynamic transmission models to simulate intervention effects. We aim to understand how the choice of network or compartmental model can influence estimates of intervention effectiveness in the short and long term for an endemic disease with susceptible and infected states in which infection, once contracted, is lifelong., Methods: We consider 4 disease models with different permutations of socially connected network versus unstructured contact (mass-action mixing) model and heterogeneous versus homogeneous disease risk. The models have susceptible and infected populations calibrated to the same long-term equilibrium disease prevalence. We consider a simple intervention with varying levels of coverage and efficacy that reduces transmission probabilities. We measure the rate of prevalence decline over the first 365 d after the intervention, long-term equilibrium prevalence, and long-term effective reproduction ratio at equilibrium., Results: Prevalence declined up to 10% faster in homogeneous risk models than heterogeneous risk models. When the disease was not eradicated, the long-term equilibrium disease prevalence was higher in mass-action mixing models than in network models by 40% or more. This difference in long-term equilibrium prevalence between network versus mass-action mixing models was greater than that of heterogeneous versus homogeneous risk models (less than 30%); network models tended to have higher effective reproduction ratios than mass-action mixing models for given combinations of intervention coverage and efficacy., Conclusions: For interventions with high efficacy and coverage, mass-action mixing models could provide a sufficient estimate of effectiveness, whereas for interventions with low efficacy and coverage, or interventions in which outcomes are measured over short time horizons, predictions from network and mass-action models diverge, highlighting the importance of sensitivity analyses on model structure., Highlights: • We calibrate 4 models-socially connected network versus unstructured contact (mass-action mixing) model and heterogeneous versus homogeneous disease risk-to 10% preintervention disease prevalence.• We measure the short- and long-term intervention effectiveness of all models using the rate of prevalence decline, long-term equilibrium disease prevalence, and effective reproduction ratio.• Generally, in the short term, prevalence declined faster in the homogeneous risk models than in the heterogeneous risk models.• Generally, in the long term, equilibrium disease prevalence was higher in the mass-action mixing models than in the network models, and the effective reproduction ratio was higher in network models than in the mass-action mixing models.

Cost-effectiveness analyses often rely on cohort state-transition models (cSTMs). The cohort trace is the primary outcome of cSTMs, which captures the proportion of the cohort in each health state over time (state occupancy). However, the cohort trace is an aggregated measure that does not capture information about the specific transitions among health states (transition dynamics). In practice, these transition dynamics are crucial in many applications, such as incorporating transition rewards or computing various epidemiological outcomes that could be used for model calibration and validation (e.g., disease incidence and lifetime risk). In this article, we propose an alternative approach to compute and store cSTMs outcomes that capture both state occupancy and transition dynamics. This approach produces a multidimensional array from which both the state occupancy and the transition dynamics can be recovered. We highlight the advantages of the multidimensional array over the traditional cohort trace and provide potential applications of the proposed approach with an example coded in R to facilitate the implementation of our method.

Background: Calibration is the process of estimating parameters of a mathematical model by matching model outputs to calibration targets. In the presence of nonidentifiability, multiple parameter sets solve the calibration problem, which may have important implications for decision making. We evaluate the implications of nonidentifiability on the optimal strategy and provide methods to check for nonidentifiability., Methods: We illustrate nonidentifiability by calibrating a 3-state Markov model of cancer relative survival (RS). We performed 2 different calibration exercises: 1) only including RS as a calibration target and 2) adding the ratio between the 2 nondeath states over time as an additional target. We used the Nelder-Mead (NM) algorithm to identify parameter sets that best matched the calibration targets. We used collinearity and likelihood profile analyses to check for nonidentifiability. We then estimated the benefit of a hypothetical treatment in terms of life expectancy gains using different, but equally good-fitting, parameter sets. We also applied collinearity analysis to a realistic model of the natural history of colorectal cancer., Results: When only RS is used as the calibration target, 2 different parameter sets yield similar maximum likelihood values. The high collinearity index and the bimodal likelihood profile on both parameters demonstrated the presence of nonidentifiability. These different, equally good-fitting parameter sets produce different estimates of the treatment effectiveness (0.67 v. 0.31 years), which could influence the optimal decision. By incorporating the additional target, the model becomes identifiable with a collinearity index of 3.5 and a unimodal likelihood profile., Conclusions: In the presence of nonidentifiability, equally likely parameter estimates might yield different conclusions. Checking for the existence of nonidentifiability and its implications should be incorporated into standard model calibration procedures.

Microsimulation models are becoming increasingly common in the field of decision modeling for health. Because microsimulation models are computationally more demanding than traditional Markov cohort models, the use of computer programming languages in their development has become more common. R is a programming language that has gained recognition within the field of decision modeling. It has the capacity to perform microsimulation models more efficiently than software commonly used for decision modeling, incorporate statistical analyses within decision models, and produce more transparent models and reproducible results. However, no clear guidance for the implementation of microsimulation models in R exists. In this tutorial, we provide a step-by-step guide to build microsimulation models in R and illustrate the use of this guide on a simple, but transferable, hypothetical decision problem. We guide the reader through the necessary steps and provide generic R code that is flexible and can be adapted for other models. We also show how this code can be extended to address more complex model structures and provide an efficient microsimulation approach that relies on vectorization solutions.

Background: To identify best-fitting input sets using model calibration, individual calibration target fits are often combined into a single goodness-of-fit (GOF) measure using a set of weights. Decisions in the calibration process, such as which weights to use, influence which sets of model inputs are identified as best-fitting, potentially leading to different health economic conclusions. We present an alternative approach to identifying best-fitting input sets based on the concept of Pareto-optimality. A set of model inputs is on the Pareto frontier if no other input set simultaneously fits all calibration targets as well or better., Methods: We demonstrate the Pareto frontier approach in the calibration of 2 models: a simple, illustrative Markov model and a previously published cost-effectiveness model of transcatheter aortic valve replacement (TAVR). For each model, we compare the input sets on the Pareto frontier to an equal number of best-fitting input sets according to 2 possible weighted-sum GOF scoring systems, and we compare the health economic conclusions arising from these different definitions of best-fitting., Results: For the simple model, outcomes evaluated over the best-fitting input sets according to the 2 weighted-sum GOF schemes were virtually nonoverlapping on the cost-effectiveness plane and resulted in very different incremental cost-effectiveness ratios ($79,300 [95% CI 72,500-87,600] v. $139,700 [95% CI 79,900-182,800] per quality-adjusted life-year [QALY] gained). Input sets on the Pareto frontier spanned both regions ($79,000 [95% CI 64,900-156,200] per QALY gained). The TAVR model yielded similar results., Conclusions: Choices in generating a summary GOF score may result in different health economic conclusions. The Pareto frontier approach eliminates the need to make these choices by using an intuitive and transparent notion of optimality as the basis for identifying best-fitting input sets., (© The Author(s) 2014.)