1. Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
- Author
-
Katharine Sherratt, Hugo Gruson, Rok Grah, Helen Johnson, Rene Niehus, Bastian Prasse, Frank Sandmann, Jannik Deuschel, Daniel Wolffram, Sam Abbott, Alexander Ullrich, Graham Gibson, Evan L Ray, Nicholas G Reich, Daniel Sheldon, Yijin Wang, Nutcha Wattanachit, Lijing Wang, Jan Trnka, Guillaume Obozinski, Tao Sun, Dorina Thanou, Loic Pottier, Ekaterina Krymova, Jan H Meinke, Maria Vittoria Barbarossa, Neele Leithäuser, Jan Mohring, Johanna Schneider, Jaroslaw Włazło, Jan Fuhrmann, Berit Lange, Isti Rodiah, Prasith Baccam, Heidi Gurung, Steven Stage, Bradley Suchoski, Jozef Budzinski, Robert Walraven, Inmaculada Villanueva, Vit Tucek, Martin Smid, Milan Zajíček, Cesar Pérez Álvarez, Borja Reina, Nikos I Bosse, Sophie R Meakin, Lauren Castro, Geoffrey Fairchild, Isaac Michaud, Dave Osthus, Pierfrancesco Alaimo Di Loro, Antonello Maruotti, Veronika Eclerová, Andrea Kraus, David Kraus, Lenka Pribylova, Bertsimas Dimitris, Michael Lingzhi Li, Soni Saksham, Jonas Dehning, Sebastian Mohr, Viola Priesemann, Grzegorz Redlarski, Benjamin Bejar, Giovanni Ardenghi, Nicola Parolini, Giovanni Ziarelli, Wolfgang Bock, Stefan Heyder, Thomas Hotz, David E Singh, Miguel Guzman-Merino, Jose L Aznarte, David Moriña, Sergio Alonso, Enric Álvarez, Daniel López, Clara Prats, Jan Pablo Burgard, Arne Rodloff, Tom Zimmermann, Alexander Kuhlmann, Janez Zibert, Fulvia Pennoni, Fabio Divino, Marti Català, Gianfranco Lovison, Paolo Giudici, Barbara Tarantino, Francesco Bartolucci, Giovanna Jona Lasinio, Marco Mingione, Alessio Farcomeni, Ajitesh Srivastava, Pablo Montero-Manso, Aniruddha Adiga, Benjamin Hurt, Bryan Lewis, Madhav Marathe, Przemyslaw Porebski, Srinivasan Venkatramanan, Rafal P Bartczuk, Filip Dreger, Anna Gambin, Krzysztof Gogolewski, Magdalena Gruziel-Słomka, Bartosz Krupa, Antoni Moszyński, Karol Niedzielewski, Jedrzej Nowosielski, Maciej Radwan, Franciszek Rakowski, Marcin Semeniuk, Ewa Szczurek, Jakub Zieliński, Jan Kisielewski, Barbara Pabjan, Holger Kirsten, Yuri Kheifetz, Markus Scholz, Przemyslaw Biecek, Marcin Bodych, Maciej Filinski, Radoslaw Idzikowski, Tyll Krueger, Tomasz Ozanski, Johannes Bracher, Sebastian Funk, Sherratt, K, Gruson, H, Grah, R, Johnson, H, Niehus, R, Prasse, B, Sandmann, F, Deuschel, J, Wolffram, D, Abbott, S, Ullrich, A, Gibson, G, L Ray, E, G Reich, N, Sheldon, D, Wang, Y, Wattanachit, N, Wang, L, Trnka, J, Obozinski, G, Sun, T, Thanou, D, Pottier, L, Krymova, E, H Meinke, J, Vittoria Barbarossa, M, Leithäuser, N, Mohring, J, Schneider, J, Włazło, J, Fuhrmann, J, Lange, B, Rodiah, I, Baccam, P, Gurung, H, Stage, S, Suchoski, B, Budzinski, J, Walraven, R, Villanueva, I, Tucek, V, Smid, M, Zajíček, M, Pérez Álvarez, C, Reina, B, I Bosse, N, R Meakin, S, Castro, L, Fairchild, G, Michaud, I, Osthus, D, Alaimo Di Loro, P, Maruotti, A, Eclerová, V, Kraus, A, Kraus, D, Pribylova, L, Dimitris, B, Lingzhi Li, M, Saksham, S, Dehning, J, Mohr, S, Priesemann, V, Redlarski, G, Bejar, B, Ardenghi, G, Parolini, N, Ziarelli, G, Bock, W, Heyder, S, Hotz, T, E Singh, D, Guzman-Merino, M, L Aznarte, J, Moriña, D, Alonso, S, Álvarez, E, López, D, Prats, C, Pablo Burgard, J, Rodloff, A, Zimmermann, T, Kuhlmann, A, Zibert, J, Pennoni, F, Divino, F, Català, M, Lovison, G, Giudici, P, Tarantino, B, Bartolucci, F, Jona Lasinio, G, Mingione, M, Farcomeni, A, Srivastava, A, Montero-Manso, P, Adiga, A, Hurt, B, Lewis, B, Marathe, M, Porebski, P, Venkatramanan, S, P Bartczuk, R, Dreger, F, Gambin, A, Gogolewski, K, Gruziel-Słomka, M, Krupa, B, Moszyński, A, Niedzielewski, K, Nowosielski, J, Radwan, M, Rakowski, F, Semeniuk, M, Szczurek, E, Zieliński, J, Kisielewski, J, Pabjan, B, Kirsten, H, Kheifetz, Y, Scholz, M, Biecek, P, Bodych, M, Filinski, M, Idzikowski, R, Krueger, T, Ozanski, T, Bracher, J, and Funk, S
- Subjects
epidemiology ,global health ,none ,General Immunology and Microbiology ,General Neuroscience ,mathematical modeling ,COVID-19 ,infectious diseases forecatsting ,General Medicine ,udc:616 ,General Biochemistry, Genetics and Molecular Biology ,COVID-19, Countries Predictions, Infectious disease, Multivariate Statistical Models, Short-term forecasts ,udc:616-036.22:519.876.5 ,SECS-S/01 - STATISTICA ,infectious diseases forecatsting, epidemiology, mathematical modeling, capacity planning, COVID-19, combining independent models, ensemble forecast ,ensemble forecast ,Settore SECS-S/01 ,napovedovanje nalezljivih bolezni, epidemiologija, matematično modeliranje, načrtovanje zmogljivosti, COVID-19, kombiniranje neodvisnih modelov, skupna napoved ,ddc:600 ,capacity planning ,combining independent models - Abstract
eLife 12, e81916 (2023). doi:10.7554/eLife.81916, Background:Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022.Methods:We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance.Results:Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models.Conclusions:Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks., Published by eLife Sciences Publications, Cambridge
- Published
- 2023