Search

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making., Competing Interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: APP report grants from Metabiota Inc outside the submitted work. J.S. and Columbia University declare partial ownership of SK Analytics. No other authors have competing interests to declare., (Copyright: This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.)

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections., (© 2023. The Author(s).)

Background: The COVID-19 Scenario Modeling Hub convened nine modeling teams to project the impact of expanding SARS-CoV-2 vaccination to children aged 5-11 years on COVID-19 burden and resilience against variant strains., Methods: Teams contributed state- and national-level weekly projections of cases, hospitalizations, and deaths in the United States from September 12, 2021 to March 12, 2022. Four scenarios covered all combinations of 1) vaccination (or not) of children aged 5-11 years (starting November 1, 2021), and 2) emergence (or not) of a variant more transmissible than the Delta variant (emerging November 15, 2021). Individual team projections were linearly pooled. The effect of childhood vaccination on overall and age-specific outcomes was estimated using meta-analyses., Findings: Assuming that a new variant would not emerge, all-age COVID-19 outcomes were projected to decrease nationally through mid-March 2022. In this setting, vaccination of children 5-11 years old was associated with reductions in projections for all-age cumulative cases (7.2%, mean incidence ratio [IR] 0.928, 95% confidence interval [CI] 0.880-0.977), hospitalizations (8.7%, mean IR 0.913, 95% CI 0.834-0.992), and deaths (9.2%, mean IR 0.908, 95% CI 0.797-1.020) compared with scenarios without childhood vaccination. Vaccine benefits increased for scenarios including a hypothesized more transmissible variant, assuming similar vaccine effectiveness. Projected relative reductions in cumulative outcomes were larger for children than for the entire population. State-level variation was observed., Interpretation: Given the scenario assumptions (defined before the emergence of Omicron), expanding vaccination to children 5-11 years old would provide measurable direct benefits, as well as indirect benefits to the all-age U.S. population, including resilience to more transmissible variants., Funding: Various (see acknowledgments)., Competing Interests: JL has served as an expert witness on cases where the likely length of the pandemic was of issue. MCR reports stock ownership in Becton Dickinson & Co., which manufactures medical equipment used in COVID-19 testing, vaccination, and treatment. JS and Columbia University disclose partial ownership of SK Analytics. JS discloses consulting for BNI. There are no other competing interests to declare.

In Spring 2021, the highly transmissible SARS-CoV-2 Delta variant began to cause increases in cases, hospitalizations, and deaths in parts of the United States. At the time, with slowed vaccination uptake, this novel variant was expected to increase the risk of pandemic resurgence in the US in summer and fall 2021. As part of the COVID-19 Scenario Modeling Hub, an ensemble of nine mechanistic models produced 6-month scenario projections for July-December 2021 for the United States. These projections estimated substantial resurgences of COVID-19 across the US resulting from the more transmissible Delta variant, projected to occur across most of the US, coinciding with school and business reopening. The scenarios revealed that reaching higher vaccine coverage in July-December 2021 reduced the size and duration of the projected resurgence substantially, with the expected impacts was largely concentrated in a subset of states with lower vaccination coverage. Despite accurate projection of COVID-19 surges occurring and timing, the magnitude was substantially underestimated 2021 by the models compared with the of the reported cases, hospitalizations, and deaths occurring during July-December, highlighting the continued challenges to predict the evolving COVID-19 pandemic. Vaccination uptake remains critical to limiting transmission and disease, particularly in states with lower vaccination coverage. Higher vaccination goals at the onset of the surge of the new variant were estimated to avert over 1.5 million cases and 21,000 deaths, although may have had even greater impacts, considering the underestimated resurgence magnitude from the model., Competing Interests: ST, CS, MQ, LM, RB, KS, EH, LC, JL, JK, HH, MK, KT, SW, LS, KR, JL, JD, JK, EL, JP, AH, DK, MC, JD, KM, XX, AP, AV, AS, PP, SV, AA, BL, BK, JO, MO, GH, BH, JC, AV, MM, SH, PB, DM, SC, RP, DJ, JT, MG, TY, SP, JH, RS, MB, MJ, CV No competing interests declared, JL has served as an expert witness on cases where the likely length of the pandemic was of issue, JS and Columbia University disclose partial ownership of SK Analytics. Discloses consulting for BNI, MR reports stock ownership in Becton Dickinson & Co, which manufactures medical equipment used in COVID testing, vaccination, and treatment

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

After a period of rapidly declining U.S. COVID-19 incidence during January-March 2021, increases occurred in several jurisdictions (1,2) despite the rapid rollout of a large-scale vaccination program. This increase coincided with the spread of more transmissible variants of SARS-CoV-2, the virus that causes COVID-19, including B.1.1.7 (1,3) and relaxation of COVID-19 prevention strategies such as those for businesses, large-scale gatherings, and educational activities. To provide long-term projections of potential trends in COVID-19 cases, hospitalizations, and deaths, COVID-19 Scenario Modeling Hub teams used a multiple-model approach comprising six models to assess the potential course of COVID-19 in the United States across four scenarios with different vaccination coverage rates and effectiveness estimates and strength and implementation of nonpharmaceutical interventions (NPIs) (public health policies, such as physical distancing and masking) over a 6-month period (April-September 2021) using data available through March 27, 2021 (4). Among the four scenarios, an accelerated decline in NPI adherence (which encapsulates NPI mandates and population behavior) was shown to undermine vaccination-related gains over the subsequent 2-3 months and, in combination with increased transmissibility of new variants, could lead to surges in cases, hospitalizations, and deaths. A sharp decline in cases was projected by July 2021, with a faster decline in the high-vaccination scenarios. High vaccination rates and compliance with public health prevention measures are essential to control the COVID-19 pandemic and to prevent surges in hospitalizations and deaths in the coming months., Competing Interests: All authors have completed and submitted the International Committee of Medical Journal Editors form for disclosure of potential conflicts of interest. Katriona Shea reports receipt of two National Science Foundation (NSF) COVID-19 RAPID awards, and a Huck Institutes of the Life Sciences Coronavirus Research Seed Grant. Rebecca Borchering reports funding from an NSF COVID-19 RAPID award. Katharine Tallaksen, Kaitlin Rainwater-Lovett, Laura Asher, Luke C. Mullany, Molly E. Gallagher, Matt Kinsey, Richard F. Obrecht, and Lauren Shin report funding from the U.S. Department of Health and Human Services (HHS), Office of the Assistant Secretary for Preparedness and Response to the Johns Hopkins Applied Physics Laboratory. Matteo Chinazzi reports grants from the National Institutes of Health (NIH), the Council of State and Territorial Epidemiologists (CSTE), and Metabiota to Northeastern University. Ana Pastore y Piontti reports funding from Metabiota, Inc. to Northeastern University and royalties from Springer Publishing. Joseph Lemaitre reports funding from the Swiss National Science Foundation, State of California, HHS, and the Department of Homeland Security (DHS). Kyra H. Grantz reports support from the California Department of Public Health, Johns Hopkins Bloomberg School of Public Health, NIH, and travel support from the World Health Organization (WHO). Elizabeth Lee and Claire Smith report support from the California Department of Public Health, Johns Hopkins Bloomberg School of Public Health, Johns Hopkins Health System, HHS, and DHS, and computing resources from Amazon Web Services, Johns Hopkins University Modeling and Policy Hub, and the Office of the Dean at the Johns Hopkins Bloomberg School of Public Health. Justin Lessler reports support from DHHS, DHS, California Institute of Technology, NIH, honorarium from the American Association for Cancer Research, personal fees for expert testimony from Paul, Weiss, Rifkind, Wharton & Garrison, LLP. Lindsay Keegan reports support from the State of California, and NIH, a University of Utah Immunology, Inflammation, and Infectious Disease Seed Grant, and a scholarship from the University of Washington Summer Institute in Statistics and Modeling of Infectious Diseases. Lucie Contamin, John Levander, Jessica Salerno, and Willem Gijsbert van Panhuis report a National Institute of General Medical Sciences grant. Ajitesh Srivastava reports a grant from the National Science Foundation. Michael C. Runge reports stock ownership in Becton Dickinson & Co., which manufactures medical equipment used in COVID testing, vaccination, and treatment. Alessandro Vespignani reports grants from NIH, NSF, WHO, CSTE, Metabiota Inc., Templeton Foundation, Scientific Interchange Foundation, Bill & Melinda Gates Foundation; royalties from Cambridge University Press, World Scientific, Springer Publishing, and Il Saggiatore; consulting fees from Human Technopole Foundation, Institute for Scientific Interchange Foundation, honorarium for lecture module at University of Washington; Scientific Advisory Board member of the Institute for Scientific Interchange Foundation, Italy, Supervisory Board member of the Human Technopole Foundation, Italy; and gifts to Northeastern University from the McGovern Foundation, the Chleck Foundation, the Sternberg Family, J. Pallotta, and Google Cloud research credits for COVID-19 from Google. Akhil Sai Peddireddy, Pyrros A. Telionis, Anil Vullikanti, Jiangzhuo Chen, Benjamin Hurt, Brian D. Klahn, Bryan Lewis, James Schlitt, Joseph Outten, Lijing Wang, Madhav Marathe, Patrick Corbett, Przemyslaw Porebski, and Srinivasan Venkatramanan report institutional support from the National Science Foundation, Expeditions, NIH, the U.S. Department of Defense, Virginia Department of Health, Virginia Department of Emergency Management, University of Virginia (internal seed grants), and Accuweather. No other potential conflicts of interest were disclosed.