Search

Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning., Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised., Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (p both <1 × 10 -9 ). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10 -14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10 -7 ., Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks., Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia., Competing Interests: Declaration of interests Dysrupt Labs, a subsidiary of SlowVoice Pty Ltd, supplied the Almanis prediction market database for this research. KM, FB, and CN are employees of Dysrupt Labs. KM, FB, CN, and ALP have stocks in Dysrupt Labs. AG, EM, MC and JPAI have no conflicts of interest., (Copyright © 2023 The Authors. Published by Elsevier B.V. All rights reserved.)