Rita R. Colwell, Luis Mier-y-Teran-Romero, Robert B. Gramacy, Linda J. Moniz, Jeremy M. Cohen, Teresa K. Yamana, David Manheim, Alicia Juarrero, Thomas Bagley, Travis C. Porco, Christopher M. Barker, Matteo Convertino, Aaron Lane, Jason Asher, Raffaele Vardavas, David L. Swerdlow, Rakibul Khan, Evan L. Ray, Jesse E. Bell, Michael A. Johansson, Justin Lessler, Xavier Rodó, Anna M. Stewart-Ibarra, Erin A. Mordecai, Antarpreet Jutla, Jason Devita, Jason R. Rohr, Sadie J. Ryan, Abraham Reddy, Melinda Moore, Sarah F Ackley, Brett M. Forshey, Terry Moschou, Osonde A. Osoba, Jeffrey Shaman, Krzysztof Sakrejda, Steven M. Babin, Nicholas G. Reich, Juli Trtanj, Ryan J. Tibshirani, Gao Jiang, Andrew M. Hebbeler, Matthew Biggerstaff, Erhan Guven, Lee Worden, Fengchen Liu, Anna L. Buczak, Brenda Rivera-Garcia, Markel García-Díez, David C. Farrow, Benjamin Baugher, Karyn M. Apfeldorf, Rachel Lowe, Dylan B. George, Richard Paul, Trevor C. Bailey, Scott Dobson, Roni Rosenfeld, Leah R. Johnson, Nick Lothian, Derek A. T. Cummings, Dhananjai M. Rao, Courtney C. Murdock, Sean M. Moore, Tridip Sardar, Daniel P. Weikel, Marilia Sá Carvalho, Jorge Rivero, Marissa Poultney, Matt Clay, Grant Osborne, Jean Paul Chretien, Alexandria C. Brown, Sangwon Hyun, Logan C. Brooks, Humberto Brito, Xi Meng, Stephen A. Lauer, Hannah E. Clapham, Yang Liu, Harold S. Margolis, Eloy Ortiz, Defense Science and Technology Organization, Johns Hopkins Bloomberg School of Public Health [Baltimore], Johns Hopkins University (JHU), University of Florida [Gainesville] (UF), Institut Català de Ciències del Clima [Barcelona] (IC3), Instituto de Fisica de Cantabria, Instituto de Física de Cantabria, Génétique fonctionnelle des maladies infectieuses - Functional Genetics of Infectious Diseases, Institut Pasteur [Paris] (IP)-Centre National de la Recherche Scientifique (CNRS), Institut Pasteur [Paris] (IP), RAND Corporation, Santa Monica, University of Minnesota [Twin Cities] (UMN), University of Minnesota System, Virginia Tech [Blacksburg], Faculté polytechnique de Mons, Université de Mons (UMons), Center for Bioinformatics and Computational Biology [Maryland] (CBCB), University of Maryland [College Park], University of Maryland System-University of Maryland System, Epidemiology and Prevention Branch, Influenza Division, Centers for Disease Control and Prevention (CDC), Centers for Disease Control and Prevention [San Juan], Centers for Disease Control and Prevention, Division of Preventive Medicine, Walter Reed Army Institute of Research, Institut Català de Ciències del Clima (IC3), Centre National de la Recherche Scientifique (CNRS)-Institut Pasteur [Paris], and Institut Pasteur [Paris]
Significance Forecasts routinely provide critical information for dangerous weather events but not yet for epidemics. Researchers develop computational models that can be used for infectious disease forecasting, but forecasts have not been broadly compared or tested. We collaboratively compared forecasts from 16 teams for 8 y of dengue epidemics in Peru and Puerto Rico. The comparison highlighted components that forecasts did well (e.g., situational awareness late in the season) and those that need more work (e.g., early season forecasts). It also identified key facets to improve forecasts, including using multiple model ensemble approaches to improve overall forecast skill. Future infectious disease forecasting work can build on these findings and this framework to improve the skill and utility of forecasts., A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecasts would be most valuable. A comparison of modeling approaches revealed that average forecast skill was lower for models including biologically meaningful data and mechanisms and that both multimodel and multiteam ensemble forecasts consistently outperformed individual model forecasts. Leveraging these insights, data, and the forecasting framework will be critical to improve forecast skill and the application of forecasts in real time for epidemic preparedness and response. Moreover, key components of this project—integration with public health needs, a common forecasting framework, shared and standardized data, and open participation—can help advance infectious disease forecasting beyond dengue.