Search

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making., Competing Interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: APP report grants from Metabiota Inc outside the submitted work. J.S. and Columbia University declare partial ownership of SK Analytics. No other authors have competing interests to declare., (Copyright: This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.)

Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022., Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1-4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models' predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models' forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models' past predictive performance., Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models' forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models' forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models' forecasts of deaths (N=763 predictions from 20 models). Across a 1-4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models., Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks., Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z)., Competing Interests: KS, HG, RG, HJ, RN, BP, FS, JD, DW, SA, AU, GG, ER, NR, DS, YW, NW, LW, JT, GO, TS, DT, LP, EK, JM, MB, NL, JM, JS, JW, JF, BL, IR, JB, RW, IV, VT, MS, MZ, CP, BR, NB, SM, LC, GF, IM, DO, PA, AM, VE, AK, DK, LP, BD, ML, SS, JD, SM, VP, GR, BB, GA, NP, GZ, WB, SH, TH, DS, MG, JA, DM, SA, EA, DL, CP, JB, AR, TZ, AK, JZ, FP, FD, MC, GL, PG, BT, FB, GJ, MM, AF, AS, PM, AA, BH, BL, MM, PP, SV, RB, FD, AG, KG, MG, BK, AM, KN, JN, MR, FR, MS, ES, JZ, JK, BP, KH, YK, MS, BP, MB, MF, RI, TK, TO, JB, SF No competing interests declared, PB, HG, SS, BS Affiliated with IEM, Inc. The author has no financial interests to declare

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

A method has been developed to significantly suppress the effect of spurious pulses on the correlation spectrum in photon correlation experiments. When the correlation experiment is operated at a low count rate (<2 true pulses per total measured correlation time), the circuit rejects an incoming pulse stream whenever more than two pulses are detected in that interval. Details of this spurious noise pulse rejection circuit are presented. This circuit has been incorporated into an existing correlation computer system as an option selected by the operator. Application of this circuit to a test case where the spurious pulse correlations were excessive compared with the signal events yielded very accurate signal correlation spectra.

Immunization of C57BL mice with one inoculum of 10(7) DBA/2-derived SL2 lymphosarcoma cells resulted in a +/- 20-fold increase in the total number of peritoneal cells. The number of macrophages showed a 10-fold increase from 3 x 10(6) (control mice) to 3.4 x 10(7) cells at day 8 after immunization. Within this macrophage population, four different cell types, based on the ultrastructural peroxidatic activity patterns, could be distinguished: exudate macrophages, resident macrophages, resident-exudate macrophages and peroxidatic-activity-negative macrophages. The number of exudate macrophages significantly increased in the peritoneal cavity after immunization: at day 8 after immunization, a peak value of 10(7) cells was observed. At the same time, there were 2.2 x 10(7) peroxidase-activity-negative macrophages present (representing the control value x 50). Significant in vitro tumoricidal activity of the isolated macrophages could not be measured until 8 days after immunization. At that time, a cytotoxicity index of 68 was reached. After immunization of the C57BL mice with 3 injections with allogeneic SL2 cells, there were no dramatic changes in the number of peritoneal cells after the last immunization. Only immediately after the last immunization was a minor increase in peroxidatic-activity-negative macrophages seen. But already at 5 days after the last immunization, the composition of the peritoneal suspension was similar to that of non-immunized mice with predominantly resident macrophages. The cytotoxicity of the peritoneal macrophages from hyperimmunized mice was constantly high during 1-15 days after the last immunization (cytotoxicity index ranged from 66-72). In order to study which type(s) of macrophage(s) (resident, exudate, resident-exudate or peroxidatic-activity-negative) is/are responsible for the cytotoxicity measured in vitro, peritoneal cell suspensions (obtained after immunization) were fractionated according to their affinity to wheat germ agglutinin (WGA) coupled to Sepharose columns. Comparison of the values of cytotoxicity measured before and after separation into "subtypes" of the macrophages revealed that the expression of cytotoxicity is not correlated with any of the "sub-types", especially when the peroxidatic activity pattern is is taken as a criterion.